Kernel analysis of deep networks
Download as PPT, PDF•1 like•658 views
This document summarizes a research paper that analyzes deep networks using kernel methods. It hypothesizes that (1) representations in higher layers of deep networks are simpler and more accurate, and (2) the network architecture controls how quickly representations are formed. The researchers used kernel principal component analysis to measure representation simplicity and accuracy at each layer of deep networks trained on MNIST and CIFAR. Their experiments found support for both hypotheses and that convolutional and pretrained networks form representations more systematically than standard multilayer perceptrons.
![Distributed Representation [1]](https://image.slidesharecdn.com/behrangmehrparavardeeplearning-150109184443-conversion-gate01/85/Kernel-analysis-of-deep-networks-4-320.jpg)
![Depth [2]](https://image.slidesharecdn.com/behrangmehrparavardeeplearning-150109184443-conversion-gate01/85/Kernel-analysis-of-deep-networks-5-320.jpg)
![Abstraction [?]](https://image.slidesharecdn.com/behrangmehrparavardeeplearning-150109184443-conversion-gate01/85/Kernel-analysis-of-deep-networks-6-320.jpg)
![Why Kernels?
1) Incorporating prior knowledge
2) Measurable simplicity and accuracy
3) Theoretical framework and convergence bounds [3]
4) Flexibility](https://image.slidesharecdn.com/behrangmehrparavardeeplearning-150109184443-conversion-gate01/85/Kernel-analysis-of-deep-networks-15-320.jpg)
![Multilayer Perceptron [4]](https://image.slidesharecdn.com/behrangmehrparavardeeplearning-150109184443-conversion-gate01/85/Kernel-analysis-of-deep-networks-25-320.jpg)
![Convolutional Neural Networks [4]](https://image.slidesharecdn.com/behrangmehrparavardeeplearning-150109184443-conversion-gate01/85/Kernel-analysis-of-deep-networks-26-320.jpg)
Kernel analysis of deep networks
- 1. Kernel Analysis of Deep Networks By: Gregoire Montavon Mikio L. Braun Klaus-Robert Muller (Technical University of Berlin) JMLR 2011 Presented by: Behrang mehrparvar (University of Houston) April 8th, 2014
- 2. Roadmap Deep Learning Goodness of Representations Measuring goodness Role of architecture
- 3. Deep Learning? Distributed representation − Less examples in regions − Capture global structure Depth − Efficient representation Abstraction − Higher-level features Flexibility − Incorporate prior knowledge
- 5. Depth [2]
- 7. Problem Specification Deep Learning is still a Black Box! Theoretical aspect − e.g. studying depth in sum-product networks Analytical arguments − e.g. analysis of depth Experimental results − e.g. performance in application domains Visualization − e.g. measuring invariance
- 8. Kernel Methods Decouples learning algorithms from data representation Kernel operator: − Measures similarity between points − All the prior knowledge of the learning problem In this paper: − Not a learning machine − Abstraction tool to model the deep network
- 9. Kernel Methods (cont.) Kernel Methods − model the deep network Used to quantify ... − the goodness of representations − the evolution of good representations
- 10. Hypothesis 1) Simpler and more accurate representation throughout the depth 2) Structure of the network (restrictions) define the speed of how representations are formed – Evolution from dist. of pixels to dist. of classes
- 11. Problem Specification Problem: Role of depth in goodness of representation Challenge: Definition and Measurement for goodness Solution: – Simplicity Dimensionality: number of kernel PCs Number of local variations – Accuracy Classification error
- 13. Method 1) Train the deep network 2) Infer the representation of each layer 3) Apply kernel PCA on each layer representations 4) Project data points on first d eigenvectors 5) Analyze the results
- 15. Why Kernels? 1) Incorporating prior knowledge 2) Measurable simplicity and accuracy 3) Theoretical framework and convergence bounds [3] 4) Flexibility
- 17. Dimensionality and Complexity (cont.)
- 18. Intuition Accuracy – Task-relevant information Simplicity – Number of allowed local variations in the inputs space – However, does not explain domain-specific regularities – Robust to number of samples • vs. number of support vectors
- 19. Effects of Kernel mapping
- 20. Experiment setup Datasets – MNIST – CIFAR Tasks – Supervised learning – Transfer learning Architectures – Multilayer perceptron (MLP) – Pretrained multilayer perceptron (PMLP) – Convolutional neural networks (CNN)
- 22. Effect of Depth (Hyp. 1)
- 23. Observation Higher layers – More accurate representations – More simple representations
- 24. Architectures Multilayer Perceptrons – No preconditioning on learning problem – Prior: NONE Pretrained Multilayer perceptrons – Better represents the underlying representation – Contains a certain part of soluton – Prior: generative model of input Convolutional Neural Networks – Prior: Spatial invariance
- 26. Convolutional Neural Networks [4]
- 27. Effect of Architecture (Hyp. 2)
- 28. Observation MNIST: – MLP: Discriminating is solved greedily – PMLP and CNN: postpone to last layers CIFAR – MLP: Doesn't discriminate till last layer – PMLP and CNN: spread it to more layers WHY?! – Good observation, but no explanation! – Hints: dataset, priors, etc. ?
- 29. Effect of Architecture (Cont.)
- 30. Observation Regularities in PMLP and CNN – Facilitate the construction of a structured solution – Controls the rate of discrimination at every level
- 31. Label Contribution of PCs
- 32. Comments Strengths – Important and interesting problem – Simple and intuitive approach – Well designed experiments – Good analysis of results Weaknesses – Too many observations • e.g. role of sigma in scale invariance – explaining observations
- 33. Future works? Experiments on Unsupervised Learning Explaining the results Analysis on biological neural systems?!
- 34. References 1) Bengio, Yoshua, and Olivier Delalleau. "On the expressive power of deep architectures." Algorithmic Learning Theory. Springer Berlin Heidelberg, 2011. 2) Poon, Hoifung, and Pedro Domingos. "Sum-product networks: A new deep architecture." Computer Vision Workshops (ICCV Workshops), 2011 IEEE International Conference on. IEEE, 2011. 3) Braun, Mikio L., Joachim M. Buhmann, and Klaus-Robert Müller. "On relevant dimensions in kernel feature spaces." The Journal of Machine Learning Research 9 (2008): 1875-1908. 4) http://deeplearning.net/
- 35. Thanks ...