Designing a neural network architecture for image recognition

1 Designing a neural network architecture for image recognition
- [Instructor] Before we start coding our image recognition neural network, let's sketch out
how it will work. This is the most basic neural network design. We feed it an image, it
passes through one or more dense layers, and then it returns an output, but this kind of
design doesn't work efficiently for images because objects can appear in lots of different
places in an image. The solution is to add one or more convolutional layers to our neural
network. These layers will help us detect patterns no matter where they appear in our
image. It can be effective to put two or more convolutional layers in a row, so in our neural
network, we'll add them in pairs. Our design so far, with two convolutional layers and the
dense layer, would work for very simple images, but there are some tricks that we can
add to our neural network to make it more efficient. The convolutional layers are
looking for patterns in our image and recording whether or not they found those patterns
in each part of our image, but we don't usually need to know exactly where in an image a
pattern was found down to the specific pixel. It's good enough to know the rough
location of where it was found. To solve this problem, we can use a technique called max
pooling. Let's look at an example. Imagine that this grid is the output of a convolutional
filter that ran over a small part of our image. It's trying to detect a particular pattern and
these numbers represent whether or not that pattern was found in the corresponding
part of the image. Let's assume that this filter is looking for patterns that look like clouds. A
zero in the grid means that the pattern wasn't found at all and a one means the area was a
strong match for the pattern. We could pass this information directly to the next layer in
our neural network, but if we can reduce the amount of information that we pass to the
next layer, it will make the neural network's job easier. The idea of max pooling is to down
sample the data by only passing on the most important bits. It works like this. First, we
divide this grid into two-by-two squares. Then, within each two-by-two square, we'll find
the largest number. If there's a tie, we'll just grab the first one. And then finally, we'll create
a new array that only saves the numbers that we selected. The idea is that we're still
capturing roughly where each pattern was found in our image, but we're doing it with 1/4
as much data. We'll get nearly the same end result, but they'll be a lot less work for the
computer to do in the following layer of the neural network. We have another trick that we
can use to make our neural network more robust, it's called dropout. One of the problems
with neural networks is that they tend to memorize the input data instead of actually
learning how to tell different objects apart. We need a way to prevent that. There's a
simple way that we can force the neural network to try really hard to learn within just
memorizing its training data. The idea is that we'll add the dropout layer between other
layers that will randomly throw away some of the data passing through it by cutting
some of the connections in the neural network. It's like going into a computer and just
randomly unplugging some cables. By randomly cutting different connections with each
training image, the neural network is forced to try harder to learn. It has to learn multiple
ways to represent the same ideas because it can't depend on any particular signal always
flowing through the neural network. It's called dropout because we're just letting some of
the data drop out of the network randomly. Dropout is an idea that might seem
counterintuitive, we're actually throwing away data to get a more accurate final result, but

in practice it works really well. We have four different kinds of layers in this neural
network. The convolutional layers add translational invariance, the max pooling layers
down sample the data, and dropout forces the neural network to learn in a more robust
way. And then finally, the dense layer maps the output of the previous layers to the output
layer so we can predict which class the image belongs to. The first three layers work really
well together, so we'll put them together into a block and we'll call the whole thing a
convolutional block. If we wanna make our neural network more powerful and able to
recognize more complex images, we can add more layers to it. But instead of just adding
layers randomly, we'll add more copies of our convolutional block. When all these layers
are working together, we'll be able to detect complex objects like dogs or cars or
airplanes. This is a very typical design for an image recognition neural network, but it's also
one of the most basic. Researchers are always experimenting with new and increasingly
complex ways of chaining together layers to improve the accuracy of their neural
networks. The latest designs involve branching pathways, shortcuts between groups of
layers and all sorts of other tricks, but they all build on these same basic ideas and this is
the approach we'll use in our code.
1.1 Exploring the CIFAR-10 data set
- [Instructor] To train neural networks to perform accurately, you need large amounts of
training data. Since it's difficult to collect thousands of training images, researches build
data sets and share them with each other. For our first image recognition project, we'll be
using the CIFAR-10 dataset. This dataset includes thousands of pictures of 10 different
kinds of objects, like airplanes, automobiles, birds, and so on. Each image in the dataset
includes a matching label so we know what kind of image it is. Using this dataset, we can
train our neural network to recognize any of these 10 different kinds of object. Before we
build an image recognition model, the first step is to look through the training data that
we are working with. We wanna check for bad or unexpected training data. Bad training
data is a very common source of problems. For example, imagine that you take millions of
photographs and ask volunteers to label them for you. This is called crowd sourcing and is
a common way to label large data sets. What if one of the labels you ask your
volunteers to use is jaguar and you have pictures of both large cats and sports cars? The
volunteers might mix up the label and sometimes use it for cats and sometimes use it for
sports cars. Because problems like this are common, it's always worth spending some time
with your training data and looking for obvious errors or problems. The images in the
CIFAR-10 dataset are only 32 pixels by 32 pixels. These are very low resolution
images. We're using them here because the lower resolution will make it possible to train
the neural network to recognize them relative quickly. With the same code we'll write, we'll
also work for larger image sizes. To make it easy for you to look through the CIFAR-10
dataset I've included some code that will display the images from the dataset on the
screen. Let's go over the PyCharm. I'm gonna open up 02 view image data set dot py. First
on line five, we have a list of the 10 different kinds of images in the dataset. Zero is plane,
one is car, and so on. Then on line 19, we'll load the dataset into memory. Keras provides
this helper function that makes it easy to access CIFAR-10. Then on line 22, we'll loop

through the first 1,000 images with a for-loop. On line 24, we grab an image from the
dataset and then on line 26 we grab that image's label. Then on line 28, we'll look up the
string name of that label from the list of labels we have at the top of the program. Then,
finally, starting on line 31, we'll use Python's Pyplot library to draw the image on the graph
and show it. Let's run the program. Right click, chose run. Here's the first image in the
dataset. It says it's a picture of a frog and if you squint, you can kinda see that it's a
frog. To see the next image in the dataset, just close this window and it will show you
another image. Try looking through several of the pictures and seeing if the labels look
correct to you. When you've got a good feel for the data, you can go back to PyCharm and
then you can click this terminate button twice to stop the program.
1.1.1 Loading an image data set
- [Instructor] To train a neural network we need a set of training images. Let's write the
code to load and pre process our training images so they're in the right format to feed into
a neural network. Let's go ahead and open up 03 loading image dataset.py. For this neural
network we'll be using the cifar10 data set. Since the cifar10 data set is used so
often, Keras provides a function for easily accessing it. Here on line eight to load the data
we'll call cifar10.loaddata. This function returns four different arrays. First it returns an x and
y array of training data. So we'll say x_train,y_train=that function call. The x array will
contain the actual images from the data set. The y array contains the matching label for
each image. The function also returns an x and y array of test data. So we'll add x_test, and
y_test. The test data is in the same format as the training data, it's just additional images
that we can use to test the neural network to make sure it's performing well. It's always
important to test a neural network with data that it didn't see during training to make sure
it actually learned how to tell the differences between images and didn't just memorize the
training data. Before we can use this data to train a neural network, we need to normalize
it. Neural networks work best when the input data are floating point values in between
zero and one. Normally images are stored as integer values for each pixel is a number
between zero and 255. So to use this data, we need to convert it from integer the floating
point and then we need to make sure all the values are between zero and one. So let's go
to line 11 and here let's convert the data to floating point values. We can do that by using
the as type function and passing in float 32. So first we'll say x train=x train.astype and
we'll pass in float32. Then we'll do exactly the same for the test array. We'll say x test = x
test astype float32. Now we'll need to scale the data so it's between zero and one. Since
we know that our pixel data is between zero and 255, we can just divide all the array values
by 255. So we can say x train = x train divide by 255. X test = x text divided by 255. When
we divide the NumPy array by a single value like this it will divide every separate array
element by 255. It's just a shorter way of writing it without having to loop through every
array and divide every element. There's one last bit of cleanup we need to do before we
can use our training data. Cifar10 provides the labels for each class as values from zero to
nine. But since we are creating a neural network with 10 outputs, we need a separate
expected value for each of those outputs. So we need to convert each label from a single
number into an array with 10 elements. In that array, one element should be set to one

and the rest set to zero. This is something you'll almost always need to do with your
training data so keras provides a helper function. It's called keras.utils.to categorical. So
let's go to line 19, and we'll say y train = keras.utils.to_categorical. To use that function you
just pass in your array with the labels which in our case is y train. And then you pass in the
number of classes it has. We know cifar10 has 10 classes. And then we can do exactly the
same thing for y test. So we'll say keras.utils.to_categorical pass in y test and 10
classes. And now we've got this data ready to use with the neural network. Let's just run
the code and double check all our code work so far. Right click choose run and it looks
good. Notice when you run your code you might get these two warning messages. That's
okay and that's expected.
2 Dense layers
- [Instructor] Now that we've loaded our data set, we're ready to create a neural network
and add the first densely connected layer to it. Let's open open up 04 dense layers.py. The
code to load the data set is already here. Starting on line 21 we're ready to add the code
to create the neural network itself. The simplest type of neural network has an input, a
densely connected layer and then an output. Let's start by creating that. First we need to
create a new neural network object in Keras. To do that, we create a new sequential
object. So we say model = sequential. The sequential api lets us create a neural network
by adding new layers to it one at a time. It's call sequential because you add each layer in
sequence and they automatically get connected together in that order. To add a new layer
we just call model.add. And then we pass on the type of layer that we want to add. Let's
create a dense layer object. This layer class takes on a few parameters. First, we need to tell
it how many nodes to include in the layer. Let's add 512 nodes to this layer. So we'll just
pass in 512. Next we need to tell it what activation function we want to use for this
layer. For a normal layer like this, a common choice is to use a rectified linear unit or relu
activation function. It's the standard choice when working with images because it works
well and is computationally efficient. So let's use that. We'll say activation=relu. And since
this is the first layer in the neural network, we need to tell it the size of the input layer. All
the images in our data set are 32 pixels by 32 pixels and have a red green and blue
channel. So for the input size we'll use 32 by 32 by 3. So we pass that in there's a
parameter called input shape. And then we pass in a list with the values 32, 32 3. And that's
everything we need for this layer. Let's go ahead and add the output layer. We'll need one
node in the output layer for each kind of object we want to detect. The cifar10 data set has
10 different kinds of objects. Since we're detecting 10 different kinds of objects, we'll
create a new dense layer with 10 nodes. So to do that we'll call model.add and we'll
create a new dense object and we know it needs 10 nodes. When doing classification with
more than one type of object, the output layer will almost always use a softmax activation
function. The softmax activation function is a special function that makes sure all the
output values from this layer add up to exactly one. The idea is that each output is a value
that represents the percent likelihood that a certain type of object was detected. And all 10
values should add up to 100 % or one. So to do that we just say activation = and we pass
in the word softmax. When we're building a neural network and adding layers to it, it's

helpful to print out a list of the layers in the neural networks so far. Let's go down to line
26 and we can do that by just calling model.summary. Let's run this code and see what the
neural network structure looks like so far. Right click choose run and let's expand this area
a little bit. Here's the output and we can see that we have two layers so far. Both are dense
layers and they're in the right order. Everything looks good so far.
2.1.1 Convolution layers
- [Instructor] So far, we've created the neural network with densely connected layers. Now
we're ready to add convolutional layers to make it better at finding patterns in
images. Let's open up 05_convolutional_layers.py. To be able to recognize images
officially, we'll add convolutional layers before our densely connected layers. Convolutional
layers are able to look for patterns in an image, no matter where the pattern appears in the
image. Let's go down to line 22, this is where we'll insert a convolutional layer. First, to add
the layer, we'll call model.add. Now there's two types of convolutional layers: 1D and
2D. Since we're working with images, we'll want to add the two dimensional convolutional
layer. For some kinds of data, like sound waves, you can use one dimensional convolutional
layers, but typically you'll be working with 2D layers. To create one, we just create a new
Conv2D object and then pass in the parameters. The first parameter is how many different
filters should be in the layer? Each filter will be capable of detecting one pattern in the
image. We'll start with 32. Next, we need to pass in the size of the window that we'll use
when creating image tiles from each image. Let's use a window size of three pixels by three
pixels. So to do that, we pass in an array of three comma three. This will split up the
original image into three by three tiles. When we do that, we have to decide what to
do with the edges of the image. If the image size isn't exactly divisible by three, we'll have
a few extra pixels left over on the edge. We can either throw that information away, or we
can add padding to the image. Padding is just extra zeros added to the edge of the image
to make the math work out. The terminology that Keras uses here is a bit confusing. If we
want to add extra padding to the image, it's called same padding. There's complex
historical reasons why researchers used the term same, but it's easier just to memorize
it. For this layer, we do want to have padding, so we'll pass in a parameter padding
equals, and the string same, and just like the normal dense layer, convolutional layers also
need an activation function. And just like dense layers, we almost always use the relu
activation function because of its efficiency. So I'll pass in activation equals relu. And that's
it for adding this layer, but there's one more tweak we need to make. Let's look at the next
line. This dense layer is no longer the first layer in the neural network, so it shouldn't
have an input shape defined anymore. Let's just cut and paste this input shape, and move
it up to the convolutional layer because it's now the first layer. To make our neural network
more powerful, let's add a few more convolutional layers the same way. First, let's add
another one with the same settings, 32 filters and a three by three window size. So we'll
say model.add, we'll pass in Conv2D, we'll say 32 filters, and the three by three window
size, and we'll also add an activation function, we'll use relu again, activation equals
relu. Now in this layer we won't have the image, so we don't need to pass in the padding
parameter. Now let's add two more layers with 64 filters each. First we'll add one with

padding, so we'll say model.add Conv2D, say 64 filters, I'll use a three by three tile size
again. I'll pass in padding equals same, and in activation function we'll use relu. And now
we'll do one more without padding, but also with 64 filters. So we can just cut and paste
this, paste it here, and just remove the padding. Alright, there's just one thing left to
do. Whenever we transition between convolutional layers and dense layers, we need to tell
Keras that we're no longer working with 2D data. To do that we need to create a flattened
layer and add it to our network. We can do that by calling model.add, and creating a new
flattened layer, and there's no parameters required for a flattened layer. Alright, if you look
down at line 35, we can see that we're printing out the summary of the neural network
structure, so let's run this code and see what it looks like. Right click and choose
run. Alright, we can see the neural network now has seven layers. We have four
convolutional layers, the flattened layer, and then our two dense layers. Notice that each
layer also has a number of parameters listed. This is the total number of weights in that
layer. There's also a total number at the bottom for the whole network. As we add more
layers that total number will keep increasing. This is the size or complexity of our neural
network. The larger the number, the longer it'll take to train and the more data we'll need
to train it. It's a good idea to keep an eye on this number as you add layers to your neural
network. As you test and refine your neural network, you might find that you can get good
results even after you remove some of your layers and reduce this number. When you can
do that, that means you'll need less powerful hardware to run your neural network, so it's
always a good goal.
3 Setting up a neural network for training
- In the previous section we wrote all the code for our neural network. Now, we're ready to
write the code that starts the training process. Open up O-one neural network training dot
p y. We already have the code here that loads the data set, and we have the code for all
the layers in the neural network and here on line 38 we've already compiled the neural
network. Now, we just need to add the code to start the training process. Let's do that
down here, on line 45. To start the training process in Kerris you call the model dot fit
function. This function takes several parameters. The first two parameters to fit are the
training data set, and the expected labels for the training data set. We already loaded
those up in our code as x training and y training. So, you can pass those in here. So, I'll
pass in x training and y training. Next, we need to pass in a batch size. The batch size is
how many images we want to feed into the network at once during training. If we set the
number too low, training will take a long time and might not ever finish. If we set the
number too high, we'll run out of memory on our computer. Typical batch sizes are
between 32 and 128 images, but feel free to experiment. For this example let's use a batch
size of 32. So, say batch size equals 32, next, we need to So, say batch size equals 32, next,
we need to decide how many times we wanna go through our training data set during the
process. One full pass through the entire training data set is called an epoch. For this
example, let's do 30 passes through the training data set. So, we'll pass in epochs equals
30. So, we'll pass in epochs equals 30. The more passes through the data we do, the more

chance the neural network has to learn; but the longer the training process will take. And
eventually you'll hit a point where doing additional training doesn't help anymore. So,
finding the right number takes some experimentation. In general, the larger your data
set, the less training passes you'll do on it. For example, for extremely large data sets with
millions of images you might only do five passes. Next, we need to tell Kerris what data we
wanna use to validate our training. This is data that the model will never see during
training, and it'll only be used to test the accuracy of the training model. When we loaded
our data set we created x test and y test, so we can use those. But pass those in as
validation data. So, we'll pass in the parameter called validation data and then, in an array
we pass in x test and y test. Finally, we need to make sure that Kerris randomizes the order
of the training data. It's very important that the neural network sees the training data
batches in random order, so that the order of the training data doesn't influence the
training. To insure that we'll pass in shuffle equals true. Shuffling is actually the default in
Kerris, but I think it's important enough that I would explicitly include it in case of changes
in a future version. Not shuffling your data can cause your model to fail to train correctly,
and that's it. We're ready to train the model, but notice that we aren't saving the results
anywhere. If we run the training process right now we'll be doing all the work and then
throwing away the results. In the next video we'll see how to save our training results, we'll
wanna do that before we run the lengthy training process.
3.1.1 Training a neural network and saving weights
- [Instructor] When we train a neural network, we wanna make sure that we save the
results, so that we can reuse the trained model later. Let's learn how to train our neural
network and save the results to a file. Open up 02 training and saving weights dot py. Here
on line eight, we've already written the code to load our dataset, and then we've coded our
neural network. And then on line 39, we've compiled it. And on line 46, we've started the
training process. But after training completes, we wanna save the trained neural network to
a file so we'll be able to use it to recognize objects and images in other programs. Let's
start that on line 56. Saving a neural network is two separate steps. First, we wanna save
the structure of the neural network itself. That includes which layers get created and the
order that they're hooked together. We could rewrite the neural network code again from
scratch each time we use it, but it's a lot easier to save the structure to a file and just load it
when we need it. Second, we wanna save the weights of the neural network. As a neural
network is trained, the weights of each node are adjusted to control how the signals flow
through the network. So by saving the weights, we're saving how the neural network
actually works. The reason we save the structure separately from the weights is because
often you'll train the same neural network multiple times with different settings or different
training datasets. It's convenient to be able to load different sets of weights using the same
neural network structure. So first, let's save the neural network structure itself. CARIS can
convert the structure of a neural network into JSON by calling the model dot to JSON
function. So we'll say 'model structure equals model dot to underscore json' Now, we just
need to write this JSON data to a text file. There's lots of ways to do this in Python, but
here is one easy way to do it using the path library. First we'll create a new path object, so

we'll say, 'f' for file equals path, and then we'll pass in the name of the file we wanna create.
So I'm just gonna call it model structure dot JSON. (typing) Then we just need to call the
right text function of the path object and pass in the data that we wanna write to the
file. So I'll do 'f' dot write text and the data that I want to write is this model structure
object, so I'll pass in model structure. Alright, now we wanna save the weights of the neural
network. This is even easier, we just need the call model that save weights and pass in the
file name. So lets go down to line 61, and I'll write model dot save weights. I'm gonna call
the file 'model weights dot h5' The data that gets saved here is in a binary format called
HDF5. The HDF5 format is designed for saving and loading large binary files efficiently. So
by convention we're using the h5 file extension to indicate the format of the file. Alright,
we're ready to train the neural network. To do that, just right click and choose 'run', and we
can watch the progress in the console here. During the training process CARIS outputs a
progress bar so we can watch what's happening. The first number on the left represents
how many samples in our training dataset have been processed. There's 50,000 total
images in our training dataset, so we can watch this number increase as training
continues. The progress bar itself represents how far along in this pass we are through the
training data. Keep in mind, we asked it to do 30 passes through the training data here on
line 50. That means that we'll do 30 complete passes through this data. So when the
progress bar is complete, that's just the first of 30 total passes. The ETA tells us how much
longer this single pass should take. The loss is the numerical representation of how wrong
our neural network is right now. The lower this number the better our neural network is
performing. We want to see this number go down during the training process. The final
number is the current accuracy. This represents how often our neural network is making
the correct prediction for the training data. We wanna see this number go up over time as
the training process continues. If the loss doesn't go down, and the accuracy doesn't go up
over time, that means there's either something wrong with the neural network design, or
that there are problems with the training data. In that case you have to go through your
code and data step by step and make sure everything looks correct. If that doesn't help, it's
possible that your dataset is too small to train your neural network, or that your neural
network doesn't have enough layers to capture the patterns in your dataset.
4 Making predictions with the trained neural network
- [Instructor] Now that we've trained our neural network, let's use it to look at new
images and make predictions. Let's open up 03_making_predictions.py. When we pass an
image through our neural network, it's going to return a likelihood for each type of object
it was trained to recognize. In order to decode those numbers into names, we need a list of
names that correspond with each number. Here, on line seven, I've already listed the
names that were used during the training process. These names correspond to the 10
types of objects that were in the CIFAR10 data set. Now, we're ready to load the neural
network. First, we need to load the structure of the network itself. One option is to write
out all the code for all the layers of the neural network again, as long as we match what
was used during training, but it's a lot easier to load the neural network structure from a
file instead. Here in the file list, we already have a file called model_structure.json. This file

contains the list of layers in our neural network, and all the details about how they were
hooked together. On line 21, we're going to load that text file into memory. We can do
that in Python by creating a new path object that represents the file that we want to
load. So we'll say f = Path(), and then as a string, in quotes, we'll pass in the name of the
file we want to load, which is model_structure.json. Then, to load the file, we can call
f.read_text() and save the results to a variable. So I'll say model_structure =
f.read_text(). Now that we have the file in memory, we need to tell Keras to rebuild the
model using that data. Keras provides a helper function to do this called
model_from_json(). So here, on line 25, we'll say model = model_from_json(), and then we'll
pass in the model_structure variable we just created. So far, we've only restored the
structure of the neural network. To restore its training as well, we need to load the weights
file we created when we trained the neural network. Here in the file list, we have a
file called model_weights.h5 that we created when we trained the model. To load it, we'll
just call model.loadweights() and pass in the filename. So here, on line 28, we'll call
model.loadweights(), and we'll pass in the filename, which is model_weights.h5. Great, now
the neural network should be ready to use. Let's find an image that we can use to try it
out. Here in the file list, I have a file called cat.png. Let's take a look. Yep, it's a picture of a
cat. Let's close it and go back to the code. On line 31, let's load this image file. To load an
image file, we can use a Keras helper function called image.load_img(). So I'll say img,
which is my image, = image.load_img(), and then I just pass in the filename to load, which
is cat.png. And finally, we need to tell it to resize the image to the size the neural network
expects. We trained this neural network with images that were 32 pixels by 32 pixels, so
that's the size we need for any images that we feed into it. To do that automatically, we
can pass in the parameter here called target_size. So we'll say target_size = , and then the
array (32, 32). Great, now that the image data is in memory, we need to convert it to a 3-D
numpy array so that we can feed it to our neural network. There's a helper function for this,
too, called image.img_to_array(). All right, so let's go down to line 34, and we'll say
image_to_test, which is the one that we'll pass into the neural network, and we'll say this =
image.img_to_array(). And then, we'll pass in the img variable we just created. Before we go
any further, we also need to normalize the image data. The image we are loading from
disk stores each pixel as a red, green, and blue value between zero and 255, but the neural
network expects an input value between zero and one. So before we can process this
image with our neural network, we need to scale the value for each pixel to a value
between zero and one. The easiest way to do this is to divide the whole array by 255, so
let's add a / 255 to the end of the line. When we divide a numpy array by a single
value, numpy will divide each individual element of the array by that value. So doing this
will scale each pixel's red, green, and blue value to a zero-to-one range. So now, this image
is ready to be processed by our trained neural network. Right now, we're only testing one
image with our neural network, but for efficiency reasons, Keras lets you pass in batches of
images at once, so you can run more than one image through the neural network at one
time. So we need to create a batch of images to pass in, even though we're only testing
this one image. Keras expects these batches as a four-dimensional array. The first
dimension is the list of images, and the other three dimensions are the image data
itself. Here's a little trick. Since we only have this one image, we can turn it into a 4-D array

by adding a new axis to it with numpy. You can do this by calling a function
called np.expand_dims() and passing in the name of the array. So on line 37, I'm going to
say list_of_images = np.expand_dims(), and I'm going to pass in image_to_test, the variable
we just created. We also need to pass in axis = 0 to tell it that the new axis is the first
dimension. This is the convention that Keras expects. Now, we have a batch of images that
we're ready to feed into the neural network and get a prediction. To do that, we'll just call
model.predict(). So on line 40, I'll say results = model.predict(), and then I'll pass in that
list_of_images we just created. The results variable will contain a list of results for each
image that we passed in. Since we only passed in one image, we can just grab the first
array index. So on line 43, I'll say single_result = results[0]. The single_result array is an
array with 10 elements. Each element represents how likely the image is to belong to each
of the object types we listed at the top of the program. Instead of returning 10 separate
numbers, let's just grab the array element with the highest value. That will tell us which
single object type was the most likely result. Let's do that on line 46 using numpy's argmax
function. So I'll say most_likely_class_index = np.argmax(), and then I'll pass in
single_result. We also want to convert this to an integer, so we'll wrap that in an int()
function. While we're here, let's also grab the likelihood value of that array index so we can
print it out later. So right below that, we'll say, class_likelihood = single_result(), and then
we'll pass in as the index the most_likely_class_index. Finally, on line 50, let's look up the
name of the object type from our list of class labels. So we'll say class_label =
class_labels(), the list we had at the top, and we'll pass in the index that we just created,
most_likely_class_index. Now, finally, on line 53, we'll just print out the results. Let's run the
program and try it out, and see if it can correctly recognize this picture of a cat. Right-click
and choose Run. Great, it predicted that our image is a cat with a likelihood of 99%. We
can also go up here to line 31 and try a different picture. So let's go up here, and
let's replace cat.png with frog.png. Frog.png is another one of the test images we have in
our folder. And let's run it again, right-click, choose Run. And great, it got this one right,
too. Feel free to try this out with your own images and see what kinds of images work
well, and what kinds of images confuse it.
4.1.1 Extracting features with a pre-trained neural network
- [Narrator] Let's use transfer learning to build an image recognition system that can
identify pictures of dogs. The first step is to build a feature extractor that can extract
training features from our images. Let's get started. First, we need some training data. I've
included some along with the example code. Let's take a look here in the training data
folder. First, I have a sub-folder called dogs. These pictures are 64 by 64 pixel images from
the image net dataset. If you're building your own image recognition system, you can use
your own pictures of whatever kind of objects you wanna recognize instead. Next, we have
a folder called "not dogs." These are various pictures of anything that's not a dog. It's
important that these pictures are as varied as possible, so that the model can learn the
difference between dogs and other types of objects. Alright, let's take a look at the
code. Open up "04_feature_extraction.py". We're gonna write the code that will use the
pretrained model to extract features from our training images and save those features to a

file. Here, starting on line eight, I've already written a code to load the list of images in
each folder. Then, on line 11, we'll create an empty array to hold the list of images. When
we process the images, we need to remember which images were dogs and which ones
were not dogs. So, on line 12, we'll create another array called labels. Each time we load an
image and put in the images array, we'll also add either a one or zero to the labels array. If
the image is a dog we'll add a one, and if the image is not a dog we'll add a zero. Then, on
line 15, we're loop through all the files in the "not dogs" folder and process each one. On
line 17 we load the image using Keras' load image helper function. This will load the image
file's contents to the memory. Then, on line 20, we convert the image data into an array
using the img to array function, and on line 23, we add that to our list of images. On line
26 we add zero to the labels array, since we know that this image is not a dog. Starting on
line 29 we'll do exactly the same thing, but this time for the dog images. The only
difference is that on line 40 we add the one to the labels array, because you know each
image is a picture of a dog. At this point we have one list with all the images and a
matching list in the same order with the labels for each image. Now we're ready to create
our training data array. On line 43 we'll create an array called x_train that will have all of
our training images. Keras expects all of our training images to be a numpy array instead
of a normal Python list. To convert the Python to a numpy array, we use the numpy array
function. So, we just say, "np.array" then we pass in the images list. And then on line 46
we'll do exactly the same thing for the labels. So we'll say, "np.array" and we pass in the
labels. To extract features we'll use the vgg16 model pretrained on the image net
dataset. This model's included with Keras. First, we need to normalize our training
dataset so all the pixel values are in the zero to one range. I've done that here on line
49 using the vgg16 preprocess_input function. Now, on line 52, we're ready to create the
neural network itself. We'll do that by creating a new vgg16 object. So, in lower case we'll
say, "vgg16 dot" and then upper case, "VGG16" to create a new object. But we also need to
pass in a few options. First, we wanna tell Keras that we wanna load the version of the
neural network that was pretrained on the image net dataset. We can do that by passing in
"weights=" and then the string, "imagenet". And since we're only using this neural
network for feature extraction, we wanna chop off the last layer of the neural
network. Since this is such a common thing to do, Keras provides a flag to tell it we wanna
do that called "include_top=False", so we'll pass in the perimeter, "include_top=False". In
Keras terminology, the top is the last layer of the neural network, so by saying
"include_top=False" we're saying we want the neural network without the last layer
attached. Finally, we need to tell it what size images we're using as training data. Our
training images are 64 pixels by 64 pixels with three color channels, one for red, one for
green, and one for blue. So, we'll pass in an input shape, say "input_shape=" and then in
the array, we're pass in "64,64,3". We're using small image sizes in this example to keep the
training time as quick as possible, but when you're building your own image recognition
systems, you can use larger sized images like 224 pixels by 224 pixels. To do that, you can
just bump up the size here. Alright, now we wanna feed all of our training images through
the neural network and capture the results. To do that on line 55, we just call the predict
function on our neural network, and pass in an array with all of our training data. So, we'll
say pretrained neural network, nn, dot predict, and we'll pass in the "x_train" variable. The

features x array will now contain the set of features that represent each of the training
images in our dataset. The last step is to save these features to disk. We can do that with a
library called joblib. It has a convenient function called dump for writing an array to
disc. I've already done that for the features on line 58 and for the labels on line 61. Alright,
let's run the program. Right click and choose run. It will take a few seconds to load all the
images and run them through our pretrained feature extractor. When this finishes it will
write up two files, x_train.dat and y_train.dat. These files contain the features and
labels that represent our training data. We'll use these features to train a new neural
network in the next section.
4.1.2 Training a new neural network with extracted features
- [Instructor] We've used the pre trained neural network to extract features from our
training images. Now we're ready to train a new neural network that uses those extracted
features. Let's open up 05 training with extracted features.py. This is the code to train a
simple neural network. The code is exactly like training any other neural network but with
two small differences. The first difference is how we load our training data. Instead of
loading raw images to train with, we're gonna load the features that we extracted with the
pre trained VGG 16 neural network. If you look at the file list on the left, you can see that
we already have our extracted features stored in a file called x train.dat and our labels
stored in a file called y train.dat. Back here on line seven and eight I've already loaded
those two files. Next, starting on line 13, we have the code to define our neural
network. The second difference is in how we define our layers. Since we use VGG 16 to
extract features from our image, this neural network has no convolutional layers. Instead it
only has the final dense layers of the neural network. These are the only layers that we'll be
retraining. Next, we'll compile the model in line 19 the same way as normal. And then on
line 16 we'll call model.fit to train the model. And then, finally, at the bottom, starting on
line 34, we'll save the train model and its weights to files. Let's run the code then train the
neural network. Right click and choose run. And notice how fast the training
completed. That took a tiny fraction of the time it would take to train a neural network
from scratch. You can see in the file list on the left that our train model is now saved in two
files, model structure.json, and model weights.h5. In the next video, we'll use our transfer
learning model to make predictions with real images.
4.1.3 Making predictions with transfer learning
- [Instructor] We've used transfer learning to create and train a neural network that can
recognize pictures of dogs. Let's see how they use that neural network to make
predictions. Open up 06_making_predictions.py. This code is exactly the same as the code
we'd use to make predictions with the standard neural network. There's just one key
change. We'll need the pre processor image with the vgg16 feature extractor. First, we can
see on line eight that we're loading the structure of the neural network. And then on line
15, we're loading the trained weights. Here on line 18, we're loading an image that we
want to test. We'll try out an image called dog.png. Let's check it out. Yep it's a picture of a
dog. Then on line 21, we're converting the image to an array and on line 24 we're turning it

to a four dimensional array so that we can feed it in that karis. So far, all the code is exactly
as it would be for any neural network. But here's the key difference. Since our neural
network was trained using features extracted from a pre trained neural network, we need
to follow the same procedure for extracting features for any image that we want to test. So
here on line 30, we need to create an instance of our pre trained neural network. This
should be exactly like the one we used to generate our training data. So we'll use the same
code here. So we'll say feature_extraction_model = vgg16. Then upper case VGG16 the
creepy object. And we'll pass on the same options. We'll say weights = in a stream image
net. We'll pass an include_top = false. And finally we need to set the input shape. So we'll
say input_shape = and then the ray 64, 64, 30. Now we need to run our image through
that pre trained neural network to extract the features that will feed into the second neural
network. We can do this by just calling the predict function on the model and saving the
result. So we'll just say feature_extraction_model.predict and then we'll pass on the
images. Great, now that we have the extracted features we can pass those in to our second
neural networks predict function to get it's final prediction for this image. So let's go down
to line 34 and we'll say results = model.predict and then we'll pass on those features we
just created. The rest of the code is exactly the same as using any other neural network. On
line 37, we just grab the first result. And then on line 40, we just print out the results. Let's
run the code and see what it predicts for this image. So right click and choose run. And it
says a picture of a dog is in fact a picture of a dog with 100 percent confidence. Let's try
another picture. Go back up here to line 18 and instead of dog.png we have another
picture called notdog.png. Let's check that one out. Yup that's not a dog. Let's close that
and let's run the code again and see what prediction we get for this image. Right click and
choose run. And it correctly predicted that this image is not a dog. Feel free to try this out
with your own images. But keep in mind that our training data is fairly small. So the
accuracy may vary. But what we just demonstrated here is really powerful. With only a few
training images, we built a program that can tell pictures of dogs apart from pictures that
aren't dogs. Only a few years ago this was science fiction. And since we used transfer
learning to do it, we're able to train the model in just seconds. Transfer learning is a very
powerful technique. Try it out on your own programs and see if you can build a new object
detection model yourself.
4.1.4 When to use an API instead of building your own solution
- [Instructor] Depending on the kind of project that you're building, sometimes it makes
more sense to use an off-the-shelf image recognition API instead of building your own
custom solution. All the major cloud vendors now provide image recognition APIs. So if
you're using cloud services from Google, Amazon, or Microsoft, you can also use their
image recognition capabilities. In addition to the products offered by the large cloud
vendors, there are also many start-ups and smaller companies that offer image recognition
APIs. All of these products will let you upload an image and get back a list of objects that
appear in that image. And best of all, using these APIs, usually only requires a few lines of
code. The downside, is that these systems have a built in list of objects that they
recognize, so you're limited to recognizing the kinds of objects that they already

understand. So when might you choose to use an API instead of training your own
machine learning model? First, if you don't have any training data, you might not have any
other choice. The APIs have their own built-in image recognition models that are already
pre-trained on many millions of images, so you don't need to do any training
yourself. Next, if you need to detect many different kinds of objects in your application, it's
often easier to use an API. Google's Cloud Vision API can detect thousands of different
objects, because they have access to a nearly unlimited amount of training data. It would
be very difficult to train your own model at that scale. Along those same lines, if you only
need to detect common types of objects, likes cars or buildings or animals, it might be
easier to use an API. These systems are pre-trained to recognize common types of
objects. Sometimes they can even give you very granular results, by telling you the specific
breed of dog if a dog is detected. Most importantly, these APIs are quick and easy to
use. So if you don't have the time or money to build your own solution, you can easily test
out an API and see if it's good enough for your project. But there are also times when
using an image recognition API just won't work for your project. If you're a position where
you have access to specialized training data, that isn't available to a company like
Microsoft or Google, it might be worth building your own model. This is also true if you're
trying to detect something very specialized, that might only apply to your industry. You're
not likely to find an off-the-shelf solution that works in very specialized cases. There are
also times when the training data is just too sensitive to share with anyone else. For
example, many medical applications train their own models, because they can't share the
underlying patient data that's used to train the model. There's also cases where the
training data might be a trade secret. But sometimes it makes sense to combine your own
model with an off-the-shelf model. In addition to basic image recognition, all of the cloud
services off their own special features. For example, Google Cloud Vision can detect the
logos of well known companies and they can detect famous landmarks in photographs. In
some cases, you might use those features in cooperation with your own custom model to
solve a larger problem. For example, you can build you own model to recognize different
types of clothing, and then use Google's API to recognize which logos appear on that
clothing. Another special case is if you need optical character recognition, or OCR. That's
where you wanna pull all the text out of a photograph. It's very difficult to build a high
quality OCR system. If you need this capability, I recommend just using Google's API for
this. You can use the API to extract text from an image, while still building your own
models to do everything else. So which vendor has the best API? There's no simple
answer. All of the vendors are constantly improving their systems with more training data
and adding new features. Depending on the type of images you're working with, one
vendor might work better than another. I recommend trying out a few different
vendors and seeing what works best for you. You can also take into consideration what
extra features the vendor offers, like logo recognition or OCR. For example, Google's
particularly good at OCR. And of course, you can always use APIs from multiple vendors. If
no one company offers all the features you need, you can combine them and use more
than one.

4.1.5 Introduction to the Google Cloud Vision API
- [Instructor] In this section we'll be using the Google Cloud Vision API for object
recognition and text extraction. Let's take a look at what Google Cloud Vision offers. The
best part about the Cloud Vision API is that you don't have to do any training yourself. You
upload an image and it gives you back results from it's pre-train model. So it's very quick
to get started. Also the pricing model is simple. You pay per 1,000 API requests and the
prices are fairly inexpensive. The catch is that each type of detection in an image counts as
one API call. So if you ask for a list of objects that appear in an image and the text that
appears in the same image that actually counts as two separate API calls. All the processing
happens on Google servers in the Cloud so you don't need any specialized hardware. You
just upload your images to Google and get back the results. Let's take a look at Google's
demo and see what kinds of data the API can extract. Open up your web browser and
go to cloud.google.com/vision. If you scroll down this webpage you'll see that Google
offers a demo. Here I have an example of a road sign. Let's drag and drop this image on
the webpage and see what Google can detect. On the first tab are the labels of the
objects that it detected in the image. We can see that the top results are
road, infrastructure, and traffic sign, which makes sense. The other results look good too,
like sky and signage. The next tab is web entities. One of the neat things Google can do is
look for webpages that had similar images and give you back results based on those
pages. So here it even guessed that the sign is from the Minnesota Department of
Transportation. The next tab, Document, is where it shows all the text that it was able to
extract. We can see that it was able to read the word road. It also looks like it detected the
other words so it's possible that we'll get more text back when we use the API. You can
also get back some document properties like dominant colors. And here on the Safe
Search tab it shows if the image contains sensitive content like violence or nudity. There
are other things that the API can do too that aren't represented here. For example it can
detect faces in the image and tell you the emotion of each person's face. Overall this is a
powerful API, but keep in mind that each tab here represents a different call to the API. So
if you want all this information about your image this would actually count as six separate
calls to the API.
4.1.6 Recognizing objects in photographs with Google Cloud Vision
- [Instructor] Alright, let's use the Cloud Vision API for image recognition. Before going any
further, make sure you've created the Google Cloud account and downloaded the
credentials file. If you aren't sure how to do that, you can review the previous
video. Alright, let's open up cloud_image_recognition.py. This file uses the Google Cloud
API to upload a file and get back a response from Google with a list of objects detected in
the image. On line seven, you can put in the name of the image file that you wanna
check. I've included the sample image to test with, so you can leave this as road_sign.jpg
for now. Let's take a look at the picture. This is a picture of a road sign on the highway. If
the API works correctly, it should come up with labels like road and sign. Alright, let's go
back to the code. On line eight is the name of the credentials file that we wanna use to
access the Cloud Vision API. You should already have a credentials.json file. If not, you can

review the previous video. On line 11 we read the credentials file into memory and then,
on line 12, we create an instance of the Google API client. Since we wanna use the Vision
API service, we pass in the string vision and v1. We also need to pass in the credentials file
in the same line. On line 15 we load the image file from disc and convert it to a base 64
encoded version. Google's API requires the images be uploaded in base 64 format. The
rest of the code here is the minimum code needed to make requests to the Google Cloud
Vision API. First, on line 20, we create an object that represents the batch request that
we're making to Google. We're required to pass in the image data to check, and then the
features that we want back. In this case, we want a list of labels of what appears in the
image so we'll pass in LABEL_DETECTION. Notice that the batch request object is an
array. In this case, we're only asking it to annotate one image, but you can pass in more
that one image in a single request if you want. Then on line 30 we create a Python request
object using the Google API library. Here we're asking it to access the image's API and then
annotate the images according to our batch request that we defined above. Then on line
33 we connect to Google and execute the request. The results will be stored in the
response object. On line 36 we check for errors, and on line 40 we get back the
results. Then finally on lines 42 and 43 we print out the results. Let's run the code and see
what happens. Right-click, choose Run, and here's what we got back. So Google says our
image is a road, infrastructure, traffic sign, sky, signage, these all look like great labels for
our image. But notice that the percentages don't add up to 100%. Unlike the custom
model we built earlier in the course, Google's model can detect multiple separate objects
in the same image. Since it's not just classifying the entire image as one type of
object, you'll get many different labels back representing separate detections. From here,
you could save these labels to a database, or use the labels to make decisions with how to
process the image. Google's done the hard work and now it's up to you to decide how you
wanna use this data in your program.
4.1.7 Next steps
- [Adam] Congratulations on completing this course. Now that you've learned how to
build image recognition models, you can try using them in your own projects. I highly
encourage you to do so. If you want to read more about image recognition, you can follow
my blog, Machine Learning Is Fun, at machinelearningisfun.com. You can also check out Py
Image Search, another great blog that covers image recognition in Python, at
pyimagesearch.com. Thanks, and feel free to follow me on Twitter in the meantime at
AGeitgey.

Designing a neural network architecture for image recognition

Designing a neural network architecture for image recognition

More Related Content

What's hot (20)

Similar to Designing a neural network architecture for image recognition (20)

Recently uploaded (20)

Designing a neural network architecture for image recognition