Autoencoder

Introduction
Autoencoders are a specific type of feed
forward neural networks where the input is the
same as the output. They compress the input into
a lower-dimensional code and then reconstruct
the output from this representation. The code is
a compact “summary” or “compression” of the
input, also called the latent-space representation.

Components of Autoencoder
An autoencoder consists of 3 components:
Encoder
Code
Decoder
Encoder compresses the input and produces
the code.
Decoder then reconstructs the input only
using this code.

Properties of Autoencoder
Data-specific
Lossy
Unsupervised

Properties of Autoencoder
Data-specific: Autoencoders are only able to
meaningfully compress data similar to what they
have been trained on.
Lossy: The output of the autoencoder will not be
exactly the same as the input, it will be a close
but degraded representation.
Unsupervised: Autoencoders are considered an
unsupervised learning technique since they don’t
need explicit labels to train on. But to be more
precise they are self-supervised because they
generate their own labels from the training data.

Architecture
Both the encoder and decoder are fully-
connected feed forward neural networks.
The number of nodes in the code layer (code
size) is a hyper parameter that we set before
training the autoencoder.

Hyperparameters
There are 4 hyperparameters that we need to
set before training an autoencoder:
Code size: number of nodes in the middle
layer. Smaller size results in more
compression.
Number of layers: the autoencoder can be as
deep as we like. In the figure above we have 2
layers in both the encoder and decoder,
without considering the input and output.

Hyperparameters
Number of nodes per layer : The layers are
stacked one after another. The number of nodes
per layer decreases with each subsequent layer of
the encoder, and increases back in the decoder.
Loss function: We either use mean squared error
(mse) or binary crossentropy. If the input values
are in the range [0, 1] then we typically use
crossentropy, otherwise we use the mean
squared error.

Types of Autoencoder :
The Four Types of Autoencoders are :-
Vanilla autoencoder
Multilayer autoencoder
Convolutional autoencoder
Regularized autoencoder

Vanilla autoencoder
The autoencoder is a three layers of a neural
net with one hidden layer. The input and
output are the same, and we learn how to
reconstruct the input.
Here, we see that we have an undercomplete
autoencoder as the hidden layer dimension
(64) is smaller than the input (784). This
constraint will impose our neural net to learn
a compressed representation of data.

Multilayer autoencoder
If one hidden layer is not enough, we can
obviously extend the autoencoder to more
hidden layers.
Now our implementation uses 3 hidden layers
instead of just one. Any of the hidden layers
can be picked as the feature representation
but we will make the network symmetrical
and use the middle-most layer.

Convolution autoencoder
We may also ask ourselves: can autoencoders be
used with Convolutions instead of Fully-
connected layers
The answer is yes and the principle is the same,
but using images (3D vectors) instead of flattened
1D vectors. The input image is downsampled to
give a latent representation of smaller
dimensions and force the autoencoder to learn a
compressed version of the images.

Regularized autoencoder
There are other ways we can constraint the
reconstruction of an autoencoder than to
impose a hidden layer of smaller dimension
than the input. Rather than limiting the model
capacity by keeping the encoder and decoder
shallow and the code size
small, regularized autoencoders use a loss
function that encourages the model to have
other properties besides the ability to copy its
input to its output.

Challenges
They are indeed pretty similar, but not exactly the
same. We can notice it more clearly in the last
digit “4”. Since this was a simple task our
autoencoder performed pretty well.
We have total control over the architecture of the
autoencoder. We can make it very powerful by
increasing the number of layers, nodes per layer
and most importantly the code size. Increasing
these hyperparameters will let the autoencoder
to learn more complex codings.

Challenges
But we should be careful to not make it too
powerful.
Over-fitting
The autoencoder will reconstruct the training
data perfectly, but it will be over fitting
without being able to generalize to new
instances, which is not what we want.

 Undercomplete
The autoencoder is said to be undercomplete.
It won’t be able to directly copy its inputs to
the output, and will be forced to learn
intelligent features. then an undercomplete
autoencoder won’t be able to recover it
perfectly.

Regularization
We would like to learn meaningful features
without altering the code’s dimensions
(Overcomplete or Undercomplete).
We usually find two types of regularized
autoencoder:
 Sparse Autoencoder
 Denoising Autoencoder

Sparse autoencoder
Sparse autoencoders are typically used to
learn features for another task such as
classification. An autoencoder that has been
regularized to be sparse must respond to
unique statistical features of the dataset it has
been trained on, rather than simply acting as
an identity function. In this way, training to
perform the copying task with a sparsity
penalty can yield a model that has learned
useful features as a by product.

Denoising Autoencoders
This way the autoencoder can’t simply
copy the input to its output because the input
also contains random noise. We are asking it
to subtract the noise and produce the
underlying meaningful data.

Conclusion
Autoencoders are a very useful
dimensionality reduction technique. They are
very popular as a teaching material in
introductory deep learning courses, most
likely due to their simplicity.

Autoencoder

More Related Content

What's hot (20)

Similar to Autoencoder (20)

Recently uploaded (20)

Autoencoder