SlideShare a Scribd company logo

Icml2012 learning hierarchies of invariant features

Z
zukun

This document discusses learning hierarchies of invariant features using convolutional neural networks. It describes how convolutional networks build hierarchical representations through multiple stacked layers that each apply normalization, filtering, non-linearity, and pooling operations to learn increasingly complex features. This architecture is inspired by the hierarchical organization of the mammalian visual cortex. The document outlines applications of convolutional networks in areas like computer vision, speech recognition, and natural language processing where they have achieved state-of-the-art performance by learning hierarchical representations from data.

TechnologyEducation
1 of 118
Downloaded 297 times
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
Learning Hierarchies Learning Hierarchies of Invariant Features of Invariant Features Yann LeCun Yann LeCun Courant Institute of Mathematical Sciences Courant Institute of Mathematical Sciences and and Center for Neural Science, Center for Neural Science, New York University New York University Yann LeCun
Challenges for Machine Learning, Vision, Signal Processing, AI, Neuroscience Challenges for Machine Learning, Vision, Signal Processing, AI, Neuroscience How can learning build a perceptual system? How do we learn representations of the perceptual world? In ML/CV/ASR/MIR: How do we learn features (not just classifiers)? With good representations, we can learn categories from just a few examples. ML has neglected the question of learning representations, relying instead on domain expertise to engineer features and kernels. Deep Learning addresses the problem of learning representations Goal 1: biologically-plausible methods for deep learning Goal 2: representation learning for computer perception Yann LeCun
Architecture of “Mainstream” Systems Architecture of “Mainstream” Systems Traditional way: handcrafted features + classifier (hand­crafted) “Simple” Trainable  Feature Extraction Classifier Mainstream Approaches to Image and Speech Recognition Low­Level Mid­Level Features Features (fixed) (unsupervised) Classifier MFCC Mix of Gaussians (Supervised) SIFT K­means HoG Sparse Coding Yann LeCun
Trainable Feature Hierarchies Trainable Feature Hierarchies Why can't we make all the modules trainable? Proposed way: hierarchy of trained features Trainable Trainable Trainable Feature Feature Classifier/ Transform Transform Predictor Learned Internal Representation Yann LeCun
The Mammalian Visual Cortex is Hierarchical The Mammalian Visual Cortex is Hierarchical The ventral (recognition) pathway in the visual cortex has multiple stages Retina - LGN - V1 - V2 - V4 - PIT - AIT .... [picture from Simon Thorpe] [Gallant & Van Essen]  Yann LeCun
Feature Transform = Feature Transform = Normalization → Filter Bank → Non-Linearity → Pooling Normalization → Filter Bank → Non-Linearity → Pooling Filter Non­ feature Filter Non­ feature Norm Norm Classifier Bank  Linear Pooling  Bank  Linear Pooling  Stacking multiple stages of [Normalization → Filter Bank → Non-Linearity → Pooling]. Normalization: variations on whitening Subtractive: average removal, high pass filtering Divisive: local contrast normalization, variance normalization Filter Bank: dimension expansion, projection on overcomplete basis Non-Linearity: sparsification, saturation, lateral inhibition.... Component-wise shrinkage or tanh, winner-takes-all Pooling: aggregation over space or feature type, subsampling 1 p 1 bX AVERAGE : K ∑ X i ; MAX : Max X i ; L p : √ X ; PROB : b log (∑ e ) i i p i i i Yann LeCun
Feature Transform = Feature Transform = Normalization → Filter Bank → Non-Linearity → Pooling Normalization → Filter Bank → Non-Linearity → Pooling Filter Non­ feature Filter Non­ feature Norm Norm Classifier Bank  Linear Pooling  Bank  Linear Pooling  Filter Bank → Non-Linearity = Non-linear embedding in high dimension Feature Pooling = contraction, dimensionality reduction, smoothing Learning the filter banks at every stage Creating a hierarchy of features Basic elements are inspired by models of the visual (and auditory) cortex Simple Cell + Complex Cell model of [Hubel and Wiesel 1962] Many “traditional” feature extraction methods are based on this SIFT, GIST, HoG, Convolutional networks..... [Fukushima 1974-1982], [LeCun 1988-now], [Poggio 2005-now], [Ng 2006-now], many others.... Yann LeCun
Basic Convolutional Network Architecture Basic Convolutional Network Architecture “Simple cells” “Complex cells” pooling  [LeCun et al. 89] Multiple  subsampling convolutions [LeCun et al. 98] Yann LeCun Retinotopic Feature Maps
Convolutional Network Architecture Convolutional Network Architecture Yann LeCun
Convolutional Network (ConvNet) Convolutional Network (ConvNet) Layer 3 256@6x6 Layer 4 Layer 1 Output Layer 2 256@1x1 64x75x75 101 input 64@14x14 83x83 9x9 9x9 10x10 pooling, convolution 6x6 pooling convolution 5x5 subsampling (4096 kernels) (64 kernels) 4x4 subsamp   Non-Linearity: shrinkage function, tanh Pooling: L2, average, max, average→tanh Training: Supervised (1988-2006), Unsupervised+Supervised (2006-now) Yann LeCun
Convolutional Network (vintage 1990) Convolutional Network (vintage 1990) filters → tanh → average-tanh → filters → tanh → average-tanh → filters → tanh Yann LeCun
“Mainstream” object recognition pipeline 2006-2010: similar to ConvNets “Mainstream” object recognition pipeline 2006-2010: similar to ConvNets Filter Non­ feature Filter Non­ feature Classifier Bank  Linearity Pooling  Bank  Linearity Pooling  Oriented Winner Histogram Pyramid SVM or K­means  Edges Takes (sum) Histogram Another  Or  All Sparse Coding (sum) Simple SIFT classifier Fixed low-level Features + unsupervised mid-level features + simple classifier Example (on Caltech 101 dataset): SIFT + Vector Quantization + Pyramid pooling + SVM: >65% [Lazebnik et al. CVPR 2006] SIFT + Local Sparse Coding Macrofeat. + Pyr/ pooling + SVM: >77% [Boureau et al. ICCV 2011] Yann LeCun
Other Applications with State-of-the-Art Performance Other Applications with State-of-the-Art Performance Traffic Sign Recognition (GTSRB) House Number Recognition (Google) German Traffic Sign Reco Bench Street View House Numbers 97.2% accuracy 94.8% accuracy Yann LeCun
ConvNet Architecture with Multi-Stage Features ConvNet Architecture with Multi-Stage Features Feature maps from all stages are pooled/subsampled and sent to the final classification layers Pooled low-level features: good for textures and local motifs High-level features: good for “gestalt” and global shape Yann LeCun [Sermanet, Chintala, LeCun ArXiv:1204.3968, 2012]
Industrial Applications of ConvNets Industrial Applications of ConvNets AT&T/Lucent/NCR Check reading, OCR, handwriting recognition (deployed 1996) NEC Intelligent vending machines and advertizing posters, cancer cell detection, automotive applications Google Face and license plate removal from StreetView images Microsoft Handwriting recognition, speech detection Orange Face detection, HCI, cell phone-based applications Startups, other companies... Yann LeCun
Fast Scene Parsing Fast Scene Parsing [Farabet, Couprie, Najman, LeCun,  ICML 2012] Yann LeCun
Labeling every pixel with the object it belongs to Labeling every pixel with the object it belongs to Would help identify obstacles, targets, landing sites, dangerous areas Would help line up depth map with edge maps Yann LeCun [Farabet et al. ICML 2012]
Scene Parsing/Labeling: ConvNet Architecture Scene Parsing/Labeling: ConvNet Architecture Each output sees a large input context: 46x46 window at full rez; 92x92 at ½ rez; 184x184 at ¼ rez [7x7conv]->[2x2pool]->[7x7conv]->[2x2pool]->[7x7conv]-> Trained supervised on fully-labeled images Categories Laplacian Level 1  Level 2 Upsampled Pyramid Features Features Level 2 Features Yann LeCun
Scene Parsing/Labeling: System Architecture Scene Parsing/Labeling: System Architecture Dense Feature Maps ConvNet Multi-Scale Pyramid (Band-pass Filtered) Original Image Yann LeCun
Method 1: majority over super-pixel regions Method 1: majority over super-pixel regions Super­pixel boundary hypetheses Majority Vote Over Superpixels Superpixel boundaries Categories aligned With region Convolutional classifier Multi­scale ConvNet boundaries Input image “soft” categories scores Features from Convolutional net Yann LeCun (d=768 per pixel) [Farabet et al. IEEE T. PAMI 2012]
Method 2: optimal cover of purity tree Method 2: optimal cover of purity tree 2-layer Distribution of Neural Categories within net Each Segment Spanning Tree From pixel Similarity graph Yann LeCun [Farabet et al. ICML 2012]
Scene Parsing/Labeling: Performance Scene Parsing/Labeling: Performance Stanford Background Dataset [Gould 1009]: 8 categories SIFT Flow dataset [Liu 2009]: 33 categories Barcelona dataset[Tighe 2010]: 170 categories. Yann LeCun [Farabet et al. IEEE T. PAMI 2012]
Scene Parsing/Labeling: Results Scene Parsing/Labeling: Results Samples from the SIFT-Flow dataset (Liu) Yann LeCun [Farabet et al. 2012]
Scene Parsing/Labeling: SIFT Flow dataset (33 categories) Scene Parsing/Labeling: SIFT Flow dataset (33 categories) Samples from the SIFT-Flow dataset (Liu) Yann LeCun [Farabet et al. ICML 2012]
Scene Parsing/Labeling: SIFT Flow dataset (33 categories) Scene Parsing/Labeling: SIFT Flow dataset (33 categories) Yann LeCun [Farabet et al. ICML 2012]
Scene Parsing/Labeling Scene Parsing/Labeling Yann LeCun [Farabet et al. ICML 2012]
Scene Parsing/Labeling Scene Parsing/Labeling Yann LeCun [Farabet et al. ICML 2012]
Scene Parsing/Labeling Scene Parsing/Labeling Yann LeCun [Farabet et al. 2012]
Scene Parsing/Labeling Scene Parsing/Labeling Yann LeCun [Farabet et al. 2012]
Scene Parsing/Labeling Scene Parsing/Labeling No post-processing Frame-by-frame ConvNet runs at 50ms/frame on Virtex-6 FPGA hardware But communicating the features over ethernet limits system perf. Yann LeCun
Scene Parsing/Labeling: Temporal Consistency Scene Parsing/Labeling: Temporal Consistency Majority Vote on Spatio-Temporal Super-Pixels Reset every second Yann LeCun
Unsupervised Unsupervised Feature Learning: Feature Learning: variations on the variations on the sparse auto-encoder theme sparse auto-encoder theme Yann LeCun
Learning Features with Unsupervised Pre-Training Learning Features with Unsupervised Pre-Training Supervised learning requires lots of labeled samples Most available data is unlabeled Models need to be large to “understand” the task But large models have many parameters and require many labeled samples Unsupervised learning can be used to pre-train the system before supervised refinement Unsupervised pre-training “consumes” degrees of freedom while placing the system in a favorable region of parameter space. Supervised refinement merely find the closest local minimum within the attractor found by unsupervised pre-training. Unsupervised feature learning through sparse/overcomplete auto-encoders With high-dimensional and sparse representations, the data manifold is “flattened” (any collection of points is flatter in higher dimension) Yann LeCun
Sparse Coding & Sparse Modeling Sparse Coding & Sparse Modeling [Olshausen & Field 1997] Sparse linear reconstruction Energy = reconstruction_error + code_prediction_error + code_sparsity i i 2 E (Y , Z )=∥Y −W d Z∥ + λ ∑ j ∣z j∣ i ∥Y −Y∥  2 WdZ ∑j . INPUT Y Z ∣z j∣ FEATURES  Inference is slow ̂ Y → Z =argmin Z E (Y , Z ) Yann LeCun
How to Speed Up Inference in aaGenerative Model? How to Speed Up Inference in Generative Model? Factor Graph with an asymmetric factor Inference Z → Y is easy Run Z through deterministic decoder, and sample Y Inference Y → Z is hard, particularly if Decoder function is many-to-one MAP: minimize sum of two factors with respect to Z Z* = argmin_z Distance[Decoder(Z), Y] + FactorB(Z) Generative Model Factor A Factor B Distance Decoder LATENT INPUT Y Z VARIABLE Yann LeCun
Idea: Train aa“simple” function to approximate the solution Idea: Train “simple” function to approximate the solution [Kavukcuoglu, Ranzato, LeCun, rejected by every conference, 2008­2009] Train a “simple” feed-forward function to predict the result of a complex optimization on the data points of interest Generative Model Factor A Factor B Distance Decoder LATENT INPUT Y Z Fast Feed­Forward Model VARIABLE Factor A' Encoder Distance 1. Find optimal Zi for all Yi; 2. Train Encoder to predict Zi from Yi Yann LeCun
Predictive Sparse Decomposition (PSD): sparse auto-encoder Predictive Sparse Decomposition (PSD): sparse auto-encoder [Kavukcuoglu, Ranzato, LeCun, 2008 → arXiv:1010.3467], Prediction the optimal code with a trained encoder Energy = reconstruction_error + code_prediction_error + code_sparsity i i 2 i 2 E Y , Z =∥Y −W d Z∥ ∥Z −g e W e ,Y ∥  ∑ j ∣z j∣ i i ge (W e , Y )=shrinkage(W e Y ) i ∥Y −Y∥  2 WdZ ∑j . INPUT Y Z ∣z j∣ FEATURES  i ge W e ,Y   2 ∥Z − Z∥ Yann LeCun
Soft Shrinkage Non-Linearity Soft Shrinkage Non-Linearity Yann LeCun
PSD: Basis Functions on MNIST PSD: Basis Functions on MNIST Basis functions (and encoder matrix) are digit parts Yann LeCun
Predictive Sparse Decomposition (PSD): Training Predictive Sparse Decomposition (PSD): Training Training on natural images patches. 12X12 256 basis functions Yann LeCun
Learned Features on natural patches: V1-like receptive fields Learned Features on natural patches: V1-like receptive fields Yann LeCun
Better Idea: Give the “right” structure to the encoder Better Idea: Give the “right” structure to the encoder [Gregor & LeCun, ICML 2010], [Bronstein et al. ICML 2012] ISTA/FISTA: iterative algorithm that converges to optimal sparse code INPUT Y We + sh () Z S Yann LeCun
LISTA: Train We and SSmatrices to give aagood approximation quickly LISTA: Train We and matrices to give good approximation quickly Think of the FISTA flow graph as a recurrent neural net where We and S are trainable parameters INPUT Y We + sh () Z S Time-Unfold the flow graph for K iterations Learn the We and S matrices with “backprop-through-time” Get the best approximate solution within K iterations Y We + sh   S + sh   S Z Yann LeCun
Learning ISTA (LISTA) vs ISTA/FISTA Learning ISTA (LISTA) vs ISTA/FISTA Yann LeCun
One Stage: filter → Shrinkage → L2 Pooling → Contrast Norm One Stage: filter → Shrinkage → L2 Pooling → Contrast Norm L2 Pooling & sub­sampling contrast normalization subtractive+divisive  Convolutions THIS IS ONE STAGE OF THE CONVNET  Shrinkage Yann LeCun
Local Contrast Normalization Local Contrast Normalization Performed on the state of every layer, including the input Subtractive Local Contrast Normalization Subtracts from every value in a feature a Gaussian-weighted average of its neighbors (high-pass filter) Divisive Local Contrast Normalization Divides every value in a layer by the standard deviation of its neighbors over space and over all feature maps Subtractive + Divisive LCN performs a kind of approximate whitening. Yann LeCun
Using PSD to Train aaHierarchy of Features Using PSD to Train Hierarchy of Features Phase 1: train first layer using PSD ∥Y i −Y∥2  WdZ ∑j . Y Z ∣z j∣ ge W e ,Y i   2 ∥Z − Z∥ FEATURES  Yann LeCun
Using PSD to Train aaHierarchy of Features Using PSD to Train Hierarchy of Features Phase 1: train first layer using PSD Phase 2: use encoder + absolute value as feature extractor Y ∣z j∣ ge W e ,Y i  FEATURES  Yann LeCun
Using PSD to Train aaHierarchy of Features Using PSD to Train Hierarchy of Features Phase 1: train first layer using PSD Phase 2: use encoder + absolute value as feature extractor Phase 3: train the second layer using PSD ∥Y i −Y∥2  WdZ ∑j . Y ∣z j∣ Y Z ∣z j∣ ge W e ,Y i  ge W e ,Y i   2 ∥Z − Z∥ FEATURES  Yann LeCun
Using PSD to Train aaHierarchy of Features Using PSD to Train Hierarchy of Features Phase 1: train first layer using PSD Phase 2: use encoder + absolute value as feature extractor Phase 3: train the second layer using PSD Phase 4: use encoder + absolute value as 2 nd feature extractor Y ∣z j∣ ∣z j∣ ge W e ,Y i  ge W e ,Y i  FEATURES  Yann LeCun
Using PSD to Train aaHierarchy of Features Using PSD to Train Hierarchy of Features Phase 1: train first layer using PSD Phase 2: use encoder + absolute value as feature extractor Phase 3: train the second layer using PSD Phase 4: use encoder + absolute value as 2 nd feature extractor Phase 5: train a supervised classifier on top Phase 6 (optional): train the entire system with supervised back-propagation Y ∣z j∣ ∣z j∣ classifier ge W e ,Y i  ge W e ,Y i  FEATURES  Yann LeCun
Using PSD Features for Object Recognition Using PSD Features for Object Recognition 64 filters on 9x9 patches trained with PSD with Linear-Sigmoid-Diagonal Encoder Yann LeCun
Multistage Hubel-Wiesel Architecture: Filters Multistage Hubel-Wiesel Architecture: Filters After PSD After supervised refinement Stage 1 Stage2 Yann LeCun
Results on Caltech101 with sigmoid non-linearity Results on Caltech101 with sigmoid non-linearity ← like HMAX model Yann LeCun
Results on Caltech101: purely supervised Results on Caltech101: purely supervised with soft-shrink, L2 pooling, contrast normalization with soft-shrink, L2 pooling, contrast normalization Supervised learning with soft-shrinkage non-linearity, L2 complex cells, and sparsity penalty on the complex cell outputs: 71% Yann LeCun
What does Local Contrast Normalization Do? What does Local Contrast Normalization Do? Original Reconstuction With LCN Reconstruction Without LCN Yann LeCun
Why Do Random Filters Work? Why Do Random Filters Work? Random Filters For Simple Optimal Cells Stimuli for each Trained Complex  Filters Cell For Simple Cells Yann LeCun
Small NORB dataset Small NORB dataset Two-stage system: error rate versus number of labeled training samples No normalization Random filters Unsup filters Sup filters Unsup+Sup filters Yann LeCun
Convolutional Sparse Coding Convolutional Sparse Coding Convolutional PSD Convolutional PSD [Kavukcuoglu, Sermanet, Boureau, Mathieu, LeCun. NIPS 2010]: convolutional PSD [Zeiler, Krishnan, Taylor, Fergus, CVPR 2010]: Deconvolutional Network [Lee, Gross, Ranganath, Ng,  ICML 2009]: Convolutional Boltzmann Machine [Norouzi, Ranjbar, Mori, CVPR 2009]:  Convolutional Boltzmann Machine [Chen, Sapiro, Dunson, Carin, Preprint 2010]: Deconvolutional Network with  automatic adjustment of code dimension. Yann LeCun
Convolutional Training Convolutional Training Problem: With patch-level training, the learning algorithm must reconstruct the entire patch with a single feature vector But when the filters are used convolutionally, neighboring feature vectors will be highly redundant Patch­level training produces lots of filters that are shifted versions of each other. Yann LeCun
Convolutional Sparse Coding Convolutional Sparse Coding Replace the dot products with dictionary element by convolutions. Input Y is a full image Each code component Zk is a feature map (an image) Each dictionary element is a convolution kernel Regular sparse coding Convolutional S.C. Y = ∑. * Zk k Wk “deconvolutional networks” [Zeiler, Taylor, Fergus CVPR 2010] Yann LeCun
Convolutional PSD: Encoder with aasoft sh() Function Convolutional PSD: Encoder with soft sh() Function Convolutional Formulation Extend sparse coding from PATCH to IMAGE PATCH based learning CONVOLUTIONAL learning Yann LeCun
Pedestrian Detection, Face Detection Pedestrian Detection, Face Detection Yann LeCun [Osadchy,Miller LeCun JMLR 2007],[Kavukcuoglu et al. NIPS 2010]
ConvNet Architecture with Multi-Stage Features ConvNet Architecture with Multi-Stage Features Feature maps from all stages are pooled/subsampled and sent to the final classification layers Pooled low-level features: good for textures and local motifs High-level features: good for “gestalt” and global shape filter+tanh Av Pooling 64 feat maps 2x2 filter+tanh Input 78x126xRGB filter+tanh L2 Pooling 22 feat maps 3x3 Yann LeCun [Sermanet, Chintala, LeCun ArXiv:1204.3968, 2012]
Pedestrian Detection (INRIA Dataset) Pedestrian Detection (INRIA Dataset) [Kavukcuoglu et al. NIPS 2010] Yann LeCun
Convolutional PSD pre-training for pedestrian detection Convolutional PSD pre-training for pedestrian detection ConvPSD pre-training improves the accuracy of pedestrian detection over purely supervised training from random initial conditions. Yann LeCun
Convolutional PSD pre-training for pedestrian detection Convolutional PSD pre-training for pedestrian detection Trained on INRIA. Tested on INRIA, Daimler, TUD-Brussles, ETH Same testing protocol as in [Dollar et al. T.PAMI 2011] Yann LeCun
Results on “Near Scale” Images (>80 pixels tall, no occlusions) Results on “Near Scale” Images (>80 pixels tall, no occlusions) INRIA Daimler p=288 p=21790 ETH TudBrussels p=804 p=508 Yann LeCun
Results on “Reasonable” Images (>50 pixels tall, few occlusions) Results on “Reasonable” Images (>50 pixels tall, few occlusions) INRIA Daimler p=288 p=21790 ETH TudBrussels p=804 p=508 Yann LeCun
Musical Genre Musical Genre Recognition Recognition Same Architecture, Different Data Same Architecture, Different Data [Henaff et al. ISMIR 2011] Yann LeCun
Convolutional PSD Features on Time-Frequency Signals Convolutional PSD Features on Time-Frequency Signals Input: “Constant Q Transform” over 46.4ms windows (1024 samples) 96 filters, with frequencies spaced every quarter tone (4 octaves) Architecture: Input: sequence of contrast-normalized CQT vectors 1: PSD features, 512 trained filters 2: shrinkage function → rectification 3: pooling over 5 seconds 4: linear SVM classifier 5: pooling of SVM categories over 30 seconds GTZAN Dataset 1000 clips, 30 second each 10 genres: blues, classical, country, disco, hiphop, jazz, metal, pop, reggae and rock. Results 84% correct classification (state of the art is at 92% with many features) Yann LeCun
Architecture: contrast norm → filters → shrink → max pooling Architecture: contrast norm → filters → shrink → max pooling contrast normalization subtractive+divisive  Linear Classifier Max Pooling (5s)  Shrinkage Filters Single­Stage Convolutional Network Training of filters: PSD (unsupervised) Yann LeCun
Constant Q Transform over 46.4 ms → Contrast Normalization Constant Q Transform over 46.4 ms → Contrast Normalization subtractive+divisive contrast normalization Yann LeCun
Convolutional PSD Features on Time-Frequency Signals Convolutional PSD Features on Time-Frequency Signals Octave-wide features full 4-octave features Minor 3rd Perfect 4th Perfect 5th Quartal chord Major triad transient Yann LeCun
PSD Features on PSD Features on Constant-Q Transform Constant-Q Transform Octave-wide features Encoder basis functions Decoder basis functions Yann LeCun
Time-Frequency Time-Frequency Features Features Octave-wide features on 8 successive acoustic vectors Almost no temporal structure in the filters! Yann LeCun
Accuracy on GTZAN dataset (small, old, etc...) Accuracy on GTZAN dataset (small, old, etc...) Accuracy: 83.4%. State of the Art: 84.3% Very fast Yann LeCun
Learning Learning Mid-Level Features Mid-Level Features [Boureau et al., CVPR 2010, ICML 2010, CVPR 2011] Yann LeCun
ConvNets and “Conventional” Vision Architectures are Similar ConvNets and “Conventional” Vision Architectures are Similar Filter Non­ feature Filter Non­ feature Classifier Bank  Linearity Pooling  Bank  Linearity Pooling  Oriented Histogram K­means Pyramid SVM with WTA  Edges (sum) Histogram Histogram (sum) Intersection SIFT kernel Can't we use the same tricks as ConvNets to train the second stage of a “conventional vision architecture? Stage 1: SIFT Stage 2: sparse coding over neighborhoods + pooling Yann LeCun
Using DL/ConvNet ideas in “conventional” recognition Using DL/ConvNet ideas in “conventional” recognition systems systems [Boureau et al. CVPR 2010] Adapting insights from ConvNets: Jointly encoding spatial neighborhoods instead of single points: increase spatial receptive fields for higher-level features Use max pooling instead of average pooling Train supervised dictionary for sparse coding This yields state-of-the-art results: 75.7% on Caltech-101 (+/-1.1%): record for single system 85.6% on 15-Scenes (+/- 0.2): record! Yann LeCun
The Competition: SIFT + Sparse-Coding + PMK-SVM The Competition: SIFT + Sparse-Coding + PMK-SVM Replacing K-means with Sparse Coding [Yang 2008] [Boureau, Bach, Ponce, LeCun 2010] Yann LeCun
Sparse Coding within Clusters Sparse Coding within Clusters Splitting the Sparse Coding into Clusters only similar things get pooled together [Boureau, et al. CVPR 2011] Yann LeCun
Learning Learning Invariant Features Invariant Features (learning complex cells) (learning complex cells) [Kavukcuoglu, Ranzato, Fergus, LeCun, CVPR 2009] [Gregor & LeCun 2010] Yann LeCun
Learning Invariant Features with L2 Group Sparsity Learning Invariant Features with L2 Group Sparsity Unsupervised PSD ignores the spatial pooling step. Could we devise a similar method that learns the pooling layer as well? Idea [Hyvarinen & Hoyer 2001]: group sparsity on pools of features Minimum number of pools must be non-zero Number of features that are on within a pool doesn't matter Pools tend to regroup similar features E (Y , Z )=∥Y −W d Z∥ + ∥Z − g e (W e ,Y )∥ + ∑ √∑ 2 2 2 Z k j k ∈P j i ∥Y −Y∥  2 WdZ ∑j . INPUT Y Z  ∑ k ∈ P  Z 2  j k FEATURES  i ge W e ,Y  ∥Z − Z∥  2 L2 norm within  each pool Yann LeCun
Learning Invariant Features with L2 Group Sparsity Learning Invariant Features with L2 Group Sparsity Idea: features are pooled in group. Sparsity: sum over groups of L2 norm of activity in group. [Hyvärinen Hoyer 2001]: “subspace ICA” decoder only, square [Welling, Hinton, Osindero NIPS 2002]: pooled product of experts encoder only, overcomplete, log student-T penalty on L2 pooling [Kavukcuoglu, Ranzato, Fergus LeCun, CVPR 2010]: Invariant PSD encoder-decoder (like PSD), overcomplete, L2 pooling [Le et al. NIPS 2011]: Reconstruction ICA Same as [Kavukcuoglu 2010] with linear encoder and tied decoder [Gregor & LeCun arXiv:1006:0448, 2010] [Le et al. ICML 2012] Locally-connect non shared (tiled) encoder-decoder L2 norm within  INPUT SIMPLE  FEATURES  each pool ∑j . Encoder only (PoE, ICA), Y Decoder Only or Encoder­Decoder (iPSD, RICA) Z ∑ j 2  k ∈ P  Z k  INVARIANT FEATURES  Yann LeCun
Pooling Similar Features using Group Sparsity Pooling Similar Features using Group Sparsity A sparse-overcomplete version of Hyvarinen's subspace ICA Decoder ensures reconstruction (unlike ICA which requires orthonogonal matrix) 1. Apply filters on a patch (with suitable non-linearity) 2. Arrange filter outputs on a 2D plane 3. square filter outputs 4. minimize sqrt of sum of blocks of squared filter outputs [Kavukcuoglu, Ranzato, Fergu, LeCun, CVPR 2009] [Jenatton, Obozinski, Bach AISTATS 2010] [Le et al. NIPS2011] Yann LeCun
Groups are local in aa2D Topographic Map Groups are local in 2D Topographic Map The filters arrange themselves spontaneously so that similar filters enter the same pool. The pooling units can be seen as complex cells Outputs of pooling units are invariant to local transformations of the input For some it's translations, for others rotations, or other transformations. Yann LeCun
Image-level training, local filters but no weight sharing Image-level training, local filters but no weight sharing Training on 115x115 images. Kernels are 15x15 (not shared across space!) [Gregor & LeCun 2010] Decoder Local receptive fields Reconstructed Input No shared weights 4x overcomplete L2 pooling Group sparsity over pools (Inferred) Code Predicted Code Input Encoder Yann LeCun
Image-level training, local filters but no weight sharing Image-level training, local filters but no weight sharing Topographic maps of continuously-varying features Local overlapping pools are invariant complex cells [Gregor & LeCun arXiv:1006.0448] (double tanh encoder) [Le et al. ICML'12] (linear encoder) Yann LeCun
Image-level training, local filters but no weight sharing Image-level training, local filters but no weight sharing Training on 115x115 images. Kernels are 15x15 (not shared across space!) Yann LeCun
K Obermayer and GG Blasdel, Journal of  Neuroscience, Vol 13, 4114­4129 (Monkey) 119x119 Image Input 100x100 Code 20x20 Receptive field size sigma=5 Michael C. Crair, et. al. The Journal of Neurophysiology  Vol. 77 No. 6 June 1997, pp. 3381­3385 (Cat)
Image-level training, local filters but no weight sharing Image-level training, local filters but no weight sharing Color indicates orientation (by fitting Gabors) Yann LeCun
Theory of Repeated [Filter Bank → L2 Pooling → Average Pooling] Theory of Repeated [Filter Bank → L2 Pooling → Average Pooling] Stéphane Mallat's “Scattering Transform”: Theory of ConvNet-like architectures [Mallat & Bruna CVPR 2011] Classification with Scattering Operators [Mallat & Bruna, arXiv:1203.1513 2012] Invariant Scattering Convolution Networks [Mallat CPAM 2012] Group Invariant Scattering Yann LeCun
Sparse Coding Using Sparse Coding Using Lateral Inhibition Lateral Inhibition [Gregor, Szlam, LeCun, NIPS 2011] Yann LeCun
Invariant Features Lateral Inhibition Invariant Features Lateral Inhibition Replace the L1 sparsity term by a lateral inhibition matrix Easy way to impose some structure on the sparsity [Gregor, Szlam, LeCun NIPS 2011] Yann LeCun
Invariant Features via Lateral Inhibition: Structured Sparsity Invariant Features via Lateral Inhibition: Structured Sparsity Each edge in the tree indicates a zero in the S matrix (no mutual inhibition) Sij is larger if two neurons are far away in the tree Yann LeCun
Yann LeCun
Yann LeCun
Yann LeCun
Invariant Features via Lateral Inhibition: Topographic Maps Invariant Features via Lateral Inhibition: Topographic Maps Non-zero values in S form a ring in a 2D topology Input patches are high-pass filtered Yann LeCun
Invariant Features via Lateral Inhibition: Topographic Maps Invariant Features via Lateral Inhibition: Topographic Maps Non-zero values in S form a ring in a 2D topology Left: no high-pass filtering of input Right: patch-level mean removal Yann LeCun
Invariant Features via Lateral Excitation: Topographic Maps Invariant Features via Lateral Excitation: Topographic Maps Short-range lateral excitation + L1 sparsity Yann LeCun
Learning What/Where Learning What/Where Features with Features with Temporal Constancy Temporal Constancy [Gregor & LeCun arXiv:1006.0448, 2010] Yann LeCun
Invariant Features through Temporal Constancy Invariant Features through Temporal Constancy Object is cross-product of object type and instantiation parameters Mapping units [Hinton 1981], capsules [Hinton 2011] small medium large Object type [Karol Gregor et al.] Object size Yann LeCun
What-Where Auto-Encoder Architecture What-Where Auto-Encoder Architecture Decoder Predicted St St­1 St­2 input W1 W1 W1 W2 t t­1 t­2 t Inferred  C 1 C 1 C 1 C 2 code C t C t­1 C t­2 C t Predicted 1 1 1 2 code f 1 1 f °W1 2 f °W f °W W  W2 W2 t t­1 t­2 Yann LeCun Encoder S S S Input
Low-Level Filters Connected to Each Complex Cell Low-Level Filters Connected to Each Complex Cell C1 (where) C2 (what) Yann LeCun
Generating from the Network Generating from the Network Input Yann LeCun
Hardware Hardware Implementations Implementations Yann LeCun
Higher End: FPGA with NeuFlow architecture Higher End: FPGA with NeuFlow architecture Now Running on Picocomputing 8x10cm high-performance FPGA board Virtex 6 LX240T: 680 MAC units, 20 neuflow tiles Full scene labeling at 20 frames/sec (50ms/frame) at 320x240 New board with Virtex­6 Yann LeCun
NewFlow: Architecture NewFlow: Architecture MEM Multi­port memory  DMA controller (DMA) [x12 on a V6 LX240T] RISC CPU, to  CPU reconfigure tiles and  data paths, at runtime global network­on­chip to  allow fast reconfiguration grid of passive processing tiles (PTs) [x20 on a Virtex6 LX240T] Yann LeCun
NewFlow: Processing Tile Architecture NewFlow: Processing Tile Architecture configurable router,  Term­by:­term  to stream data in  streaming  operators  and out of the tile, to  (MUL,DIV,ADD,SU neighbors or DMA  B,MAX) ports [x20] [x8,2 per tile] configurable piece­wise  linear  or quadratic  mapper [x4] configurable bank of  full 1/2D  parallel convolver  FIFOs , for stream  with 100 MAC units buffering, up to 10kB  per PT [Virtex6 LX240T] [x4] [x8] Yann LeCun
NewFlow ASIC: 2.5x5 mm, 45nm, 0.6Watts, 160GOPS NewFlow ASIC: 2.5x5 mm, 45nm, 0.6Watts, 160GOPS Collaboration NYU-Purdue (Eugenio Culurciello's group) Suitable for vision-enabled embedded and mobile devices Status: waiting for first samples end of 2012 Yann LeCun Pham, Jelaca, Farabet, Martini, LeCun, Culurciello 2012] 
NewFlow: Performance NewFlow: Performance Intel neuFlow neuFlow nVidia  neuFlow nVidia I7 4 cores Virtex4 Virtex 6 GT335m IBM 45nm GTX480 Peak GOP/sec 40 40 160 182 160 1350 Actual GOP/sec 12 37 147 54 147 294 FPS 14 46 182 67 182 374 Power  (W) 50 10 10 30 0.6 220 Embed? (GOP/s/W) 0.24 3.7 14.7 1.8 245 1.34 Yann LeCun
Software Platforms: Software Platforms: Torch7 Torch7 http://www.torch.ch http://www.torch.ch [Collobert, Kavukcuoglu, Farabet 2011] Yann LeCun
Torch7 Torch7 ML/Vision development environment Collaboration between IDIAP, NYU, and NEC Labs Uses the Lua interpreter/compiler as a front-end Very simple and efficient scripting language Ultra simple and natural syntax It's like Scheme underneath, but with a “normal” syntax. Lua is widely used in the video game and web industries Torch7 extend Lua with Numerical, ML, Vision libraries Powerful Vector/Matrix/Tensor Engine (developed by us) Interfaces to numerical libraries, OpenCV, etc Back-ends for SSE, OpenMP, CUDA, ARM-Neon,... Qt-based GUI and graphics Open source: http://www.torch.ch/ First released in early 2012 Yann LeCun
MATRIX and VECTOR Operations Torch7 Torch7 > M = torch.Tensor(2,2):fill(2) > N = torch.Tensor(2,4):fill(3) ML/Vision development environment > x = torch.Tensor(2):fill(4) As simple as Matlab, but faster and > y = torch.Tensor(2):fill(5) better designed as a language. > = x*y -- dot product 40 2D CONVOLUTION > = M*x --- matrix-vector x = torch.rand(100,100) k = torch.rand(10,10) 16 res1 = torch.conv2(x,k) 16 [torch.Tensor of dimension 2] TRAINING A NEURAL NET mlp = nn.Sequential() mlp:add( nn.Linear(10, 25) ) -- 10 input, 25 hidden units mlp:add( nn.Tanh() ) -- some hyperbolic tangent transfer function mlp:add( nn.Linear(25, 1) ) -- 1 output criterion = nn.MSECriterion() -- Mean Squared Error criterion trainer = nn.StochasticGradient(mlp, criterion) trainer:train(dataset) -- train using some examples Yann LeCun
Acknowledgements Acknowledgements Camille Couprie Karol Gregor Y-Lan Boureau Clément Farabet Kevin Jarrett Arthur Szlam Koray Kavukcuoglu Marc'Aurelio Ranzato Rob Fergus Pierre Sermanet Laurent Najman (ESIEE) Eugenio Culurciello (Purdue) Yann LeCun
The End The End Yann LeCun

More Related Content

What's hot (20)

PDF
Artificial neural networks
stellajoseph
 
PPTX
Introduction to deep learning
Junaid Bhat
 
PDF
Deep Learning and Tensorflow Implementation(딥러닝, 텐서플로우, 파이썬, CNN)_Myungyon Ki...
Myungyon Kim
 
PDF
A Brief Introduction on Recurrent Neural Network and Its Application
Xiaohu ZHU
 
PDF
Convolutional Neural Networks (CNN)
Gaurav Mittal
 
PPTX
CNN Tutorial
Sungjoon Choi
 
PPTX
Deep neural networks
Si Haem
 
PDF
2010 deep learning and unsupervised feature learning
Van Thanh
 
PPTX
Image captioning
Muhammad Zbeedat
 
PDF
From neural networks to deep learning
Viet-Trung TRAN
 
PPT
Deep Learning
Roshan Chettri
 
PPTX
Image Segmentation Using Deep Learning : A survey
NUPUR YADAV
 
PDF
Fundamental of deep learning
Stanley Wang
 
PPTX
Unsupervised Feature Learning
Amgad Muhammad
 
PDF
Scene classification using Convolutional Neural Networks - Jayani Withanawasam
WithTheBest
 
PPTX
TypeScript and Deep Learning
Oswald Campesato
 
PPTX
Deep Learning via Semi-Supervised Embedding （第 7 回 Deep Learning 勉強会資料; 大澤）
Ohsawa Goodfellow
 
PDF
Deep learning frameworks v0.40
Jessica Willis
 
PPTX
Introduction to CNN
Shuai Zhang
 
PPTX
Autoencoders for image_classification
Cenk Bircanoğlu
 
Artificial neural networks
stellajoseph
 
Introduction to deep learning
Junaid Bhat
 
Deep Learning and Tensorflow Implementation(딥러닝, 텐서플로우, 파이썬, CNN)_Myungyon Ki...
Myungyon Kim
 
A Brief Introduction on Recurrent Neural Network and Its Application
Xiaohu ZHU
 
Convolutional Neural Networks (CNN)
Gaurav Mittal
 
CNN Tutorial
Sungjoon Choi
 
Deep neural networks
Si Haem
 
2010 deep learning and unsupervised feature learning
Van Thanh
 
Image captioning
Muhammad Zbeedat
 
From neural networks to deep learning
Viet-Trung TRAN
 
Deep Learning
Roshan Chettri
 
Image Segmentation Using Deep Learning : A survey
NUPUR YADAV
 
Fundamental of deep learning
Stanley Wang
 
Unsupervised Feature Learning
Amgad Muhammad
 
Scene classification using Convolutional Neural Networks - Jayani Withanawasam
WithTheBest
 
TypeScript and Deep Learning
Oswald Campesato
 
Deep Learning via Semi-Supervised Embedding （第 7 回 Deep Learning 勉強会資料; 大澤）
Ohsawa Goodfellow
 
Deep learning frameworks v0.40
Jessica Willis
 
Introduction to CNN
Shuai Zhang
 
Autoencoders for image_classification
Cenk Bircanoğlu
 

Similar to Icml2012 learning hierarchies of invariant features (20)

PDF
Fcv learn le_cun
zukun
 
PDF
What's Wrong With Deep Learning?
Philip Zheng
 
PDF
Lecun 20060816-ciar-02-deep learning for generic object recognition
zukun
 
PDF
Multiple Kernel Learning based Approach to Representation and Feature Selecti...
ICAC09
 
PDF
A tutorial on deep learning at icml 2013
Philip Zheng
 
PPTX
Visualization of Deep Learning
YaminiAlapati1
 
PPTX
Elettronica: Multimedia Information Processing in Smart Environments by Aless...
Codemotion
 
PPTX
Deep learning from a novice perspective
Anirban Santara
 
PPTX
Introduction to Deep Learning for Image Analysis at Strata NYC, Sep 2015
Turi, Inc.
 
PDF
Machine Learning in Computer Vision
butest
 
PDF
Machine Learning in Computer Vision
butest
 
PDF
Introduction to Machine Learning
butest
 
PDF
Introduction to Machine Learning
butest
 
PPTX
The Shanghai-Hongkong Team at MediaEval2012: Violent Scene Detection Using Tr...
MediaEval2012
 
PDF
[系列活動] 一日搞懂生成式對抗網路
台灣資料科學年會
 
PPT
Taxonomy-Based Glyph Design
Eamonn Maguire
 
PPTX
Artificial Intelligence, Machine Learning and Deep Learning
Sujit Pal
 
PPT
lecun-01.ppt
VenkyChinna8
 
PPTX
Convolutional neural networks 이론과 응용
홍배 김
 
PDF
Fcv bio cv_simoncelli
zukun
 
Fcv learn le_cun
zukun
 
What's Wrong With Deep Learning?
Philip Zheng
 
Lecun 20060816-ciar-02-deep learning for generic object recognition
zukun
 
Multiple Kernel Learning based Approach to Representation and Feature Selecti...
ICAC09
 
A tutorial on deep learning at icml 2013
Philip Zheng
 
Visualization of Deep Learning
YaminiAlapati1
 
Elettronica: Multimedia Information Processing in Smart Environments by Aless...
Codemotion
 
Deep learning from a novice perspective
Anirban Santara
 
Introduction to Deep Learning for Image Analysis at Strata NYC, Sep 2015
Turi, Inc.
 
Machine Learning in Computer Vision
butest
 
Machine Learning in Computer Vision
butest
 
Introduction to Machine Learning
butest
 
Introduction to Machine Learning
butest
 
The Shanghai-Hongkong Team at MediaEval2012: Violent Scene Detection Using Tr...
MediaEval2012
 
[系列活動] 一日搞懂生成式對抗網路
台灣資料科學年會
 
Taxonomy-Based Glyph Design
Eamonn Maguire
 
Artificial Intelligence, Machine Learning and Deep Learning
Sujit Pal
 
lecun-01.ppt
VenkyChinna8
 
Convolutional neural networks 이론과 응용
홍배 김
 
Fcv bio cv_simoncelli
zukun
 
Ad

More from zukun (20)

PDF
My lyn tutorial 2009
zukun
 
PDF
ETHZ CV2012: Tutorial openCV
zukun
 
PDF
ETHZ CV2012: Information
zukun
 
PDF
Siwei lyu: natural image statistics
zukun
 
PDF
Lecture9 camera calibration
zukun
 
PDF
Brunelli 2008: template matching techniques in computer vision
zukun
 
PDF
Modern features-part-4-evaluation
zukun
 
PDF
Modern features-part-3-software
zukun
 
PDF
Modern features-part-2-descriptors
zukun
 
PDF
Modern features-part-1-detectors
zukun
 
PDF
Modern features-part-0-intro
zukun
 
PDF
Lecture 02 internet video search
zukun
 
PDF
Lecture 01 internet video search
zukun
 
PDF
Lecture 03 internet video search
zukun
 
PDF
Icml2012 tutorial representation_learning
zukun
 
PPT
Advances in discrete energy minimisation for computer vision
zukun
 
PDF
Gephi tutorial: quick start
zukun
 
PDF
EM algorithm and its application in probabilistic latent semantic analysis
zukun
 
PDF
Object recognition with pictorial structures
zukun
 
PDF
Iccv2011 learning spatiotemporal graphs of human activities
zukun
 
My lyn tutorial 2009
zukun
 
ETHZ CV2012: Tutorial openCV
zukun
 
ETHZ CV2012: Information
zukun
 
Siwei lyu: natural image statistics
zukun
 
Lecture9 camera calibration
zukun
 
Brunelli 2008: template matching techniques in computer vision
zukun
 
Modern features-part-4-evaluation
zukun
 
Modern features-part-3-software
zukun
 
Modern features-part-2-descriptors
zukun
 
Modern features-part-1-detectors
zukun
 
Modern features-part-0-intro
zukun
 
Lecture 02 internet video search
zukun
 
Lecture 01 internet video search
zukun
 
Lecture 03 internet video search
zukun
 
Icml2012 tutorial representation_learning
zukun
 
Advances in discrete energy minimisation for computer vision
zukun
 
Gephi tutorial: quick start
zukun
 
EM algorithm and its application in probabilistic latent semantic analysis
zukun
 
Object recognition with pictorial structures
zukun
 
Iccv2011 learning spatiotemporal graphs of human activities
zukun
 
Ad

Recently uploaded (20)

PDF
Timothy Rottach - Ramp up on AI Use Cases, from Vector Search to AI Agents wi...
AWS Chicago
 
PDF
Blockchain Transactions Explained For Everyone
CIFDAQ
 
PDF
Newgen 2022-Forrester Newgen TEI_13 05 2022-The-Total-Economic-Impact-Newgen-...
darshakparmar
 
PDF
Transcript: New from BookNet Canada for 2025: BNC BiblioShare - Tech Forum 2025
BookNet Canada
 
PDF
Fl Studio 24.2.2 Build 4597 Crack for Windows Free Download 2025
faizk77g
 
PDF
Python basic programing language for automation
DanialHabibi2
 
PDF
CIFDAQ Market Insights for July 7th 2025
CIFDAQ
 
PDF
Using FME to Develop Self-Service CAD Applications for a Major UK Police Force
Safe Software
 
PDF
Bitcoin for Millennials podcast with Bram, Power Laws of Bitcoin
Stephen Perrenod
 
PPTX
Webinar: Introduction to LF Energy EVerest
DanBrown980551
 
PPTX
From Sci-Fi to Reality: Exploring AI Evolution
Svetlana Meissner
 
PPTX
COMPARISON OF RASTER ANALYSIS TOOLS OF QGIS AND ARCGIS
Sharanya Sarkar
 
PPTX
AI Penetration Testing Essentials: A Cybersecurity Guide for 2025
defencerabbit Team
 
PDF
HubSpot Main Hub: A Unified Growth Platform
Jaswinder Singh
 
PDF
HCIP-Data Center Facility Deployment V2.0 Training Material (Without Remarks ...
mcastillo49
 
PPTX
Building Search Using OpenSearch: Limitations and Workarounds
Sease
 
PPTX
Q2 FY26 Tableau User Group Leader Quarterly Call
lward7
 
PDF
The Builder’s Playbook - 2025 State of AI Report.pdf
jeroen339954
 
PDF
NewMind AI - Journal 100 Insights After The 100th Issue
NewMind AI
 
PPTX
UiPath Academic Alliance Educator Panels: Session 2 - Business Analyst Content
DianaGray10
 
Timothy Rottach - Ramp up on AI Use Cases, from Vector Search to AI Agents wi...
AWS Chicago
 
Blockchain Transactions Explained For Everyone
CIFDAQ
 
Newgen 2022-Forrester Newgen TEI_13 05 2022-The-Total-Economic-Impact-Newgen-...
darshakparmar
 
Transcript: New from BookNet Canada for 2025: BNC BiblioShare - Tech Forum 2025
BookNet Canada
 
Fl Studio 24.2.2 Build 4597 Crack for Windows Free Download 2025
faizk77g
 
Python basic programing language for automation
DanialHabibi2
 
CIFDAQ Market Insights for July 7th 2025
CIFDAQ
 
Using FME to Develop Self-Service CAD Applications for a Major UK Police Force
Safe Software
 
Bitcoin for Millennials podcast with Bram, Power Laws of Bitcoin
Stephen Perrenod
 
Webinar: Introduction to LF Energy EVerest
DanBrown980551
 
From Sci-Fi to Reality: Exploring AI Evolution
Svetlana Meissner
 
COMPARISON OF RASTER ANALYSIS TOOLS OF QGIS AND ARCGIS
Sharanya Sarkar
 
AI Penetration Testing Essentials: A Cybersecurity Guide for 2025
defencerabbit Team
 
HubSpot Main Hub: A Unified Growth Platform
Jaswinder Singh
 
HCIP-Data Center Facility Deployment V2.0 Training Material (Without Remarks ...
mcastillo49
 
Building Search Using OpenSearch: Limitations and Workarounds
Sease
 
Q2 FY26 Tableau User Group Leader Quarterly Call
lward7
 
The Builder’s Playbook - 2025 State of AI Report.pdf
jeroen339954
 
NewMind AI - Journal 100 Insights After The 100th Issue
NewMind AI
 
UiPath Academic Alliance Educator Panels: Session 2 - Business Analyst Content
DianaGray10
 

Icml2012 learning hierarchies of invariant features

  • 1. Learning Hierarchies Learning Hierarchies of Invariant Features of Invariant Features Yann LeCun Yann LeCun Courant Institute of Mathematical Sciences Courant Institute of Mathematical Sciences and and Center for Neural Science, Center for Neural Science, New York University New York University Yann LeCun
  • 2. Challenges for Machine Learning, Vision, Signal Processing, AI, Neuroscience Challenges for Machine Learning, Vision, Signal Processing, AI, Neuroscience How can learning build a perceptual system? How do we learn representations of the perceptual world? In ML/CV/ASR/MIR: How do we learn features (not just classifiers)? With good representations, we can learn categories from just a few examples. ML has neglected the question of learning representations, relying instead on domain expertise to engineer features and kernels. Deep Learning addresses the problem of learning representations Goal 1: biologically-plausible methods for deep learning Goal 2: representation learning for computer perception Yann LeCun
  • 3. Architecture of “Mainstream” Systems Architecture of “Mainstream” Systems Traditional way: handcrafted features + classifier (hand­crafted) “Simple” Trainable  Feature Extraction Classifier Mainstream Approaches to Image and Speech Recognition Low­Level Mid­Level Features Features (fixed) (unsupervised) Classifier MFCC Mix of Gaussians (Supervised) SIFT K­means HoG Sparse Coding Yann LeCun
  • 4. Trainable Feature Hierarchies Trainable Feature Hierarchies Why can't we make all the modules trainable? Proposed way: hierarchy of trained features Trainable Trainable Trainable Feature Feature Classifier/ Transform Transform Predictor Learned Internal Representation Yann LeCun
  • 5. The Mammalian Visual Cortex is Hierarchical The Mammalian Visual Cortex is Hierarchical The ventral (recognition) pathway in the visual cortex has multiple stages Retina - LGN - V1 - V2 - V4 - PIT - AIT .... [picture from Simon Thorpe] [Gallant & Van Essen]  Yann LeCun
  • 6. Feature Transform = Feature Transform = Normalization → Filter Bank → Non-Linearity → Pooling Normalization → Filter Bank → Non-Linearity → Pooling Filter Non­ feature Filter Non­ feature Norm Norm Classifier Bank  Linear Pooling  Bank  Linear Pooling  Stacking multiple stages of [Normalization → Filter Bank → Non-Linearity → Pooling]. Normalization: variations on whitening Subtractive: average removal, high pass filtering Divisive: local contrast normalization, variance normalization Filter Bank: dimension expansion, projection on overcomplete basis Non-Linearity: sparsification, saturation, lateral inhibition.... Component-wise shrinkage or tanh, winner-takes-all Pooling: aggregation over space or feature type, subsampling 1 p 1 bX AVERAGE : K ∑ X i ; MAX : Max X i ; L p : √ X ; PROB : b log (∑ e ) i i p i i i Yann LeCun
  • 7. Feature Transform = Feature Transform = Normalization → Filter Bank → Non-Linearity → Pooling Normalization → Filter Bank → Non-Linearity → Pooling Filter Non­ feature Filter Non­ feature Norm Norm Classifier Bank  Linear Pooling  Bank  Linear Pooling  Filter Bank → Non-Linearity = Non-linear embedding in high dimension Feature Pooling = contraction, dimensionality reduction, smoothing Learning the filter banks at every stage Creating a hierarchy of features Basic elements are inspired by models of the visual (and auditory) cortex Simple Cell + Complex Cell model of [Hubel and Wiesel 1962] Many “traditional” feature extraction methods are based on this SIFT, GIST, HoG, Convolutional networks..... [Fukushima 1974-1982], [LeCun 1988-now], [Poggio 2005-now], [Ng 2006-now], many others.... Yann LeCun
  • 8. Basic Convolutional Network Architecture Basic Convolutional Network Architecture “Simple cells” “Complex cells” pooling  [LeCun et al. 89] Multiple  subsampling convolutions [LeCun et al. 98] Yann LeCun Retinotopic Feature Maps
  • 9. Convolutional Network Architecture Convolutional Network Architecture Yann LeCun
  • 10. Convolutional Network (ConvNet) Convolutional Network (ConvNet) Layer 3 256@6x6 Layer 4 Layer 1 Output Layer 2 256@1x1 64x75x75 101 input 64@14x14 83x83 9x9 9x9 10x10 pooling, convolution 6x6 pooling convolution 5x5 subsampling (4096 kernels) (64 kernels) 4x4 subsamp   Non-Linearity: shrinkage function, tanh Pooling: L2, average, max, average→tanh Training: Supervised (1988-2006), Unsupervised+Supervised (2006-now) Yann LeCun
  • 11. Convolutional Network (vintage 1990) Convolutional Network (vintage 1990) filters → tanh → average-tanh → filters → tanh → average-tanh → filters → tanh Yann LeCun
  • 12. “Mainstream” object recognition pipeline 2006-2010: similar to ConvNets “Mainstream” object recognition pipeline 2006-2010: similar to ConvNets Filter Non­ feature Filter Non­ feature Classifier Bank  Linearity Pooling  Bank  Linearity Pooling  Oriented Winner Histogram Pyramid SVM or K­means  Edges Takes (sum) Histogram Another  Or  All Sparse Coding (sum) Simple SIFT classifier Fixed low-level Features + unsupervised mid-level features + simple classifier Example (on Caltech 101 dataset): SIFT + Vector Quantization + Pyramid pooling + SVM: >65% [Lazebnik et al. CVPR 2006] SIFT + Local Sparse Coding Macrofeat. + Pyr/ pooling + SVM: >77% [Boureau et al. ICCV 2011] Yann LeCun
  • 13. Other Applications with State-of-the-Art Performance Other Applications with State-of-the-Art Performance Traffic Sign Recognition (GTSRB) House Number Recognition (Google) German Traffic Sign Reco Bench Street View House Numbers 97.2% accuracy 94.8% accuracy Yann LeCun
  • 14. ConvNet Architecture with Multi-Stage Features ConvNet Architecture with Multi-Stage Features Feature maps from all stages are pooled/subsampled and sent to the final classification layers Pooled low-level features: good for textures and local motifs High-level features: good for “gestalt” and global shape Yann LeCun [Sermanet, Chintala, LeCun ArXiv:1204.3968, 2012]
  • 15. Industrial Applications of ConvNets Industrial Applications of ConvNets AT&T/Lucent/NCR Check reading, OCR, handwriting recognition (deployed 1996) NEC Intelligent vending machines and advertizing posters, cancer cell detection, automotive applications Google Face and license plate removal from StreetView images Microsoft Handwriting recognition, speech detection Orange Face detection, HCI, cell phone-based applications Startups, other companies... Yann LeCun
  • 16. Fast Scene Parsing Fast Scene Parsing [Farabet, Couprie, Najman, LeCun,  ICML 2012] Yann LeCun
  • 17. Labeling every pixel with the object it belongs to Labeling every pixel with the object it belongs to Would help identify obstacles, targets, landing sites, dangerous areas Would help line up depth map with edge maps Yann LeCun [Farabet et al. ICML 2012]
  • 18. Scene Parsing/Labeling: ConvNet Architecture Scene Parsing/Labeling: ConvNet Architecture Each output sees a large input context: 46x46 window at full rez; 92x92 at ½ rez; 184x184 at ¼ rez [7x7conv]->[2x2pool]->[7x7conv]->[2x2pool]->[7x7conv]-> Trained supervised on fully-labeled images Categories Laplacian Level 1  Level 2 Upsampled Pyramid Features Features Level 2 Features Yann LeCun
  • 19. Scene Parsing/Labeling: System Architecture Scene Parsing/Labeling: System Architecture Dense Feature Maps ConvNet Multi-Scale Pyramid (Band-pass Filtered) Original Image Yann LeCun
  • 20. Method 1: majority over super-pixel regions Method 1: majority over super-pixel regions Super­pixel boundary hypetheses Majority Vote Over Superpixels Superpixel boundaries Categories aligned With region Convolutional classifier Multi­scale ConvNet boundaries Input image “soft” categories scores Features from Convolutional net Yann LeCun (d=768 per pixel) [Farabet et al. IEEE T. PAMI 2012]
  • 21. Method 2: optimal cover of purity tree Method 2: optimal cover of purity tree 2-layer Distribution of Neural Categories within net Each Segment Spanning Tree From pixel Similarity graph Yann LeCun [Farabet et al. ICML 2012]
  • 22. Scene Parsing/Labeling: Performance Scene Parsing/Labeling: Performance Stanford Background Dataset [Gould 1009]: 8 categories SIFT Flow dataset [Liu 2009]: 33 categories Barcelona dataset[Tighe 2010]: 170 categories. Yann LeCun [Farabet et al. IEEE T. PAMI 2012]
  • 23. Scene Parsing/Labeling: Results Scene Parsing/Labeling: Results Samples from the SIFT-Flow dataset (Liu) Yann LeCun [Farabet et al. 2012]
  • 24. Scene Parsing/Labeling: SIFT Flow dataset (33 categories) Scene Parsing/Labeling: SIFT Flow dataset (33 categories) Samples from the SIFT-Flow dataset (Liu) Yann LeCun [Farabet et al. ICML 2012]
  • 25. Scene Parsing/Labeling: SIFT Flow dataset (33 categories) Scene Parsing/Labeling: SIFT Flow dataset (33 categories) Yann LeCun [Farabet et al. ICML 2012]
  • 26. Scene Parsing/Labeling Scene Parsing/Labeling Yann LeCun [Farabet et al. ICML 2012]
  • 27. Scene Parsing/Labeling Scene Parsing/Labeling Yann LeCun [Farabet et al. ICML 2012]
  • 28. Scene Parsing/Labeling Scene Parsing/Labeling Yann LeCun [Farabet et al. 2012]
  • 29. Scene Parsing/Labeling Scene Parsing/Labeling Yann LeCun [Farabet et al. 2012]
  • 30. Scene Parsing/Labeling Scene Parsing/Labeling No post-processing Frame-by-frame ConvNet runs at 50ms/frame on Virtex-6 FPGA hardware But communicating the features over ethernet limits system perf. Yann LeCun
  • 31. Scene Parsing/Labeling: Temporal Consistency Scene Parsing/Labeling: Temporal Consistency Majority Vote on Spatio-Temporal Super-Pixels Reset every second Yann LeCun
  • 32. Unsupervised Unsupervised Feature Learning: Feature Learning: variations on the variations on the sparse auto-encoder theme sparse auto-encoder theme Yann LeCun
  • 33. Learning Features with Unsupervised Pre-Training Learning Features with Unsupervised Pre-Training Supervised learning requires lots of labeled samples Most available data is unlabeled Models need to be large to “understand” the task But large models have many parameters and require many labeled samples Unsupervised learning can be used to pre-train the system before supervised refinement Unsupervised pre-training “consumes” degrees of freedom while placing the system in a favorable region of parameter space. Supervised refinement merely find the closest local minimum within the attractor found by unsupervised pre-training. Unsupervised feature learning through sparse/overcomplete auto-encoders With high-dimensional and sparse representations, the data manifold is “flattened” (any collection of points is flatter in higher dimension) Yann LeCun
  • 34. Sparse Coding & Sparse Modeling Sparse Coding & Sparse Modeling [Olshausen & Field 1997] Sparse linear reconstruction Energy = reconstruction_error + code_prediction_error + code_sparsity i i 2 E (Y , Z )=∥Y −W d Z∥ + λ ∑ j ∣z j∣ i ∥Y −Y∥  2 WdZ ∑j . INPUT Y Z ∣z j∣ FEATURES  Inference is slow ̂ Y → Z =argmin Z E (Y , Z ) Yann LeCun
  • 35. How to Speed Up Inference in aaGenerative Model? How to Speed Up Inference in Generative Model? Factor Graph with an asymmetric factor Inference Z → Y is easy Run Z through deterministic decoder, and sample Y Inference Y → Z is hard, particularly if Decoder function is many-to-one MAP: minimize sum of two factors with respect to Z Z* = argmin_z Distance[Decoder(Z), Y] + FactorB(Z) Generative Model Factor A Factor B Distance Decoder LATENT INPUT Y Z VARIABLE Yann LeCun
  • 36. Idea: Train aa“simple” function to approximate the solution Idea: Train “simple” function to approximate the solution [Kavukcuoglu, Ranzato, LeCun, rejected by every conference, 2008­2009] Train a “simple” feed-forward function to predict the result of a complex optimization on the data points of interest Generative Model Factor A Factor B Distance Decoder LATENT INPUT Y Z Fast Feed­Forward Model VARIABLE Factor A' Encoder Distance 1. Find optimal Zi for all Yi; 2. Train Encoder to predict Zi from Yi Yann LeCun
  • 37. Predictive Sparse Decomposition (PSD): sparse auto-encoder Predictive Sparse Decomposition (PSD): sparse auto-encoder [Kavukcuoglu, Ranzato, LeCun, 2008 → arXiv:1010.3467], Prediction the optimal code with a trained encoder Energy = reconstruction_error + code_prediction_error + code_sparsity i i 2 i 2 E Y , Z =∥Y −W d Z∥ ∥Z −g e W e ,Y ∥  ∑ j ∣z j∣ i i ge (W e , Y )=shrinkage(W e Y ) i ∥Y −Y∥  2 WdZ ∑j . INPUT Y Z ∣z j∣ FEATURES  i ge W e ,Y   2 ∥Z − Z∥ Yann LeCun
  • 38. Soft Shrinkage Non-Linearity Soft Shrinkage Non-Linearity Yann LeCun
  • 39. PSD: Basis Functions on MNIST PSD: Basis Functions on MNIST Basis functions (and encoder matrix) are digit parts Yann LeCun
  • 40. Predictive Sparse Decomposition (PSD): Training Predictive Sparse Decomposition (PSD): Training Training on natural images patches. 12X12 256 basis functions Yann LeCun
  • 41. Learned Features on natural patches: V1-like receptive fields Learned Features on natural patches: V1-like receptive fields Yann LeCun
  • 42. Better Idea: Give the “right” structure to the encoder Better Idea: Give the “right” structure to the encoder [Gregor & LeCun, ICML 2010], [Bronstein et al. ICML 2012] ISTA/FISTA: iterative algorithm that converges to optimal sparse code INPUT Y We + sh () Z S Yann LeCun
  • 43. LISTA: Train We and SSmatrices to give aagood approximation quickly LISTA: Train We and matrices to give good approximation quickly Think of the FISTA flow graph as a recurrent neural net where We and S are trainable parameters INPUT Y We + sh () Z S Time-Unfold the flow graph for K iterations Learn the We and S matrices with “backprop-through-time” Get the best approximate solution within K iterations Y We + sh   S + sh   S Z Yann LeCun
  • 44. Learning ISTA (LISTA) vs ISTA/FISTA Learning ISTA (LISTA) vs ISTA/FISTA Yann LeCun
  • 45. One Stage: filter → Shrinkage → L2 Pooling → Contrast Norm One Stage: filter → Shrinkage → L2 Pooling → Contrast Norm L2 Pooling & sub­sampling contrast normalization subtractive+divisive  Convolutions THIS IS ONE STAGE OF THE CONVNET  Shrinkage Yann LeCun
  • 46. Local Contrast Normalization Local Contrast Normalization Performed on the state of every layer, including the input Subtractive Local Contrast Normalization Subtracts from every value in a feature a Gaussian-weighted average of its neighbors (high-pass filter) Divisive Local Contrast Normalization Divides every value in a layer by the standard deviation of its neighbors over space and over all feature maps Subtractive + Divisive LCN performs a kind of approximate whitening. Yann LeCun
  • 47. Using PSD to Train aaHierarchy of Features Using PSD to Train Hierarchy of Features Phase 1: train first layer using PSD ∥Y i −Y∥2  WdZ ∑j . Y Z ∣z j∣ ge W e ,Y i   2 ∥Z − Z∥ FEATURES  Yann LeCun
  • 48. Using PSD to Train aaHierarchy of Features Using PSD to Train Hierarchy of Features Phase 1: train first layer using PSD Phase 2: use encoder + absolute value as feature extractor Y ∣z j∣ ge W e ,Y i  FEATURES  Yann LeCun
  • 49. Using PSD to Train aaHierarchy of Features Using PSD to Train Hierarchy of Features Phase 1: train first layer using PSD Phase 2: use encoder + absolute value as feature extractor Phase 3: train the second layer using PSD ∥Y i −Y∥2  WdZ ∑j . Y ∣z j∣ Y Z ∣z j∣ ge W e ,Y i  ge W e ,Y i   2 ∥Z − Z∥ FEATURES  Yann LeCun
  • 50. Using PSD to Train aaHierarchy of Features Using PSD to Train Hierarchy of Features Phase 1: train first layer using PSD Phase 2: use encoder + absolute value as feature extractor Phase 3: train the second layer using PSD Phase 4: use encoder + absolute value as 2 nd feature extractor Y ∣z j∣ ∣z j∣ ge W e ,Y i  ge W e ,Y i  FEATURES  Yann LeCun
  • 51. Using PSD to Train aaHierarchy of Features Using PSD to Train Hierarchy of Features Phase 1: train first layer using PSD Phase 2: use encoder + absolute value as feature extractor Phase 3: train the second layer using PSD Phase 4: use encoder + absolute value as 2 nd feature extractor Phase 5: train a supervised classifier on top Phase 6 (optional): train the entire system with supervised back-propagation Y ∣z j∣ ∣z j∣ classifier ge W e ,Y i  ge W e ,Y i  FEATURES  Yann LeCun
  • 52. Using PSD Features for Object Recognition Using PSD Features for Object Recognition 64 filters on 9x9 patches trained with PSD with Linear-Sigmoid-Diagonal Encoder Yann LeCun
  • 53. Multistage Hubel-Wiesel Architecture: Filters Multistage Hubel-Wiesel Architecture: Filters After PSD After supervised refinement Stage 1 Stage2 Yann LeCun
  • 54. Results on Caltech101 with sigmoid non-linearity Results on Caltech101 with sigmoid non-linearity ← like HMAX model Yann LeCun
  • 55. Results on Caltech101: purely supervised Results on Caltech101: purely supervised with soft-shrink, L2 pooling, contrast normalization with soft-shrink, L2 pooling, contrast normalization Supervised learning with soft-shrinkage non-linearity, L2 complex cells, and sparsity penalty on the complex cell outputs: 71% Yann LeCun
  • 56. What does Local Contrast Normalization Do? What does Local Contrast Normalization Do? Original Reconstuction With LCN Reconstruction Without LCN Yann LeCun
  • 57. Why Do Random Filters Work? Why Do Random Filters Work? Random Filters For Simple Optimal Cells Stimuli for each Trained Complex  Filters Cell For Simple Cells Yann LeCun
  • 58. Small NORB dataset Small NORB dataset Two-stage system: error rate versus number of labeled training samples No normalization Random filters Unsup filters Sup filters Unsup+Sup filters Yann LeCun
  • 59. Convolutional Sparse Coding Convolutional Sparse Coding Convolutional PSD Convolutional PSD [Kavukcuoglu, Sermanet, Boureau, Mathieu, LeCun. NIPS 2010]: convolutional PSD [Zeiler, Krishnan, Taylor, Fergus, CVPR 2010]: Deconvolutional Network [Lee, Gross, Ranganath, Ng,  ICML 2009]: Convolutional Boltzmann Machine [Norouzi, Ranjbar, Mori, CVPR 2009]:  Convolutional Boltzmann Machine [Chen, Sapiro, Dunson, Carin, Preprint 2010]: Deconvolutional Network with  automatic adjustment of code dimension. Yann LeCun
  • 60. Convolutional Training Convolutional Training Problem: With patch-level training, the learning algorithm must reconstruct the entire patch with a single feature vector But when the filters are used convolutionally, neighboring feature vectors will be highly redundant Patch­level training produces lots of filters that are shifted versions of each other. Yann LeCun
  • 61. Convolutional Sparse Coding Convolutional Sparse Coding Replace the dot products with dictionary element by convolutions. Input Y is a full image Each code component Zk is a feature map (an image) Each dictionary element is a convolution kernel Regular sparse coding Convolutional S.C. Y = ∑. * Zk k Wk “deconvolutional networks” [Zeiler, Taylor, Fergus CVPR 2010] Yann LeCun
  • 62. Convolutional PSD: Encoder with aasoft sh() Function Convolutional PSD: Encoder with soft sh() Function Convolutional Formulation Extend sparse coding from PATCH to IMAGE PATCH based learning CONVOLUTIONAL learning Yann LeCun
  • 63. Pedestrian Detection, Face Detection Pedestrian Detection, Face Detection Yann LeCun [Osadchy,Miller LeCun JMLR 2007],[Kavukcuoglu et al. NIPS 2010]
  • 64. ConvNet Architecture with Multi-Stage Features ConvNet Architecture with Multi-Stage Features Feature maps from all stages are pooled/subsampled and sent to the final classification layers Pooled low-level features: good for textures and local motifs High-level features: good for “gestalt” and global shape filter+tanh Av Pooling 64 feat maps 2x2 filter+tanh Input 78x126xRGB filter+tanh L2 Pooling 22 feat maps 3x3 Yann LeCun [Sermanet, Chintala, LeCun ArXiv:1204.3968, 2012]
  • 65. Pedestrian Detection (INRIA Dataset) Pedestrian Detection (INRIA Dataset) [Kavukcuoglu et al. NIPS 2010] Yann LeCun
  • 66. Convolutional PSD pre-training for pedestrian detection Convolutional PSD pre-training for pedestrian detection ConvPSD pre-training improves the accuracy of pedestrian detection over purely supervised training from random initial conditions. Yann LeCun
  • 67. Convolutional PSD pre-training for pedestrian detection Convolutional PSD pre-training for pedestrian detection Trained on INRIA. Tested on INRIA, Daimler, TUD-Brussles, ETH Same testing protocol as in [Dollar et al. T.PAMI 2011] Yann LeCun
  • 68. Results on “Near Scale” Images (>80 pixels tall, no occlusions) Results on “Near Scale” Images (>80 pixels tall, no occlusions) INRIA Daimler p=288 p=21790 ETH TudBrussels p=804 p=508 Yann LeCun
  • 69. Results on “Reasonable” Images (>50 pixels tall, few occlusions) Results on “Reasonable” Images (>50 pixels tall, few occlusions) INRIA Daimler p=288 p=21790 ETH TudBrussels p=804 p=508 Yann LeCun
  • 70. Musical Genre Musical Genre Recognition Recognition Same Architecture, Different Data Same Architecture, Different Data [Henaff et al. ISMIR 2011] Yann LeCun
  • 71. Convolutional PSD Features on Time-Frequency Signals Convolutional PSD Features on Time-Frequency Signals Input: “Constant Q Transform” over 46.4ms windows (1024 samples) 96 filters, with frequencies spaced every quarter tone (4 octaves) Architecture: Input: sequence of contrast-normalized CQT vectors 1: PSD features, 512 trained filters 2: shrinkage function → rectification 3: pooling over 5 seconds 4: linear SVM classifier 5: pooling of SVM categories over 30 seconds GTZAN Dataset 1000 clips, 30 second each 10 genres: blues, classical, country, disco, hiphop, jazz, metal, pop, reggae and rock. Results 84% correct classification (state of the art is at 92% with many features) Yann LeCun
  • 72. Architecture: contrast norm → filters → shrink → max pooling Architecture: contrast norm → filters → shrink → max pooling contrast normalization subtractive+divisive  Linear Classifier Max Pooling (5s)  Shrinkage Filters Single­Stage Convolutional Network Training of filters: PSD (unsupervised) Yann LeCun
  • 73. Constant Q Transform over 46.4 ms → Contrast Normalization Constant Q Transform over 46.4 ms → Contrast Normalization subtractive+divisive contrast normalization Yann LeCun
  • 74. Convolutional PSD Features on Time-Frequency Signals Convolutional PSD Features on Time-Frequency Signals Octave-wide features full 4-octave features Minor 3rd Perfect 4th Perfect 5th Quartal chord Major triad transient Yann LeCun
  • 75. PSD Features on PSD Features on Constant-Q Transform Constant-Q Transform Octave-wide features Encoder basis functions Decoder basis functions Yann LeCun
  • 76. Time-Frequency Time-Frequency Features Features Octave-wide features on 8 successive acoustic vectors Almost no temporal structure in the filters! Yann LeCun
  • 77. Accuracy on GTZAN dataset (small, old, etc...) Accuracy on GTZAN dataset (small, old, etc...) Accuracy: 83.4%. State of the Art: 84.3% Very fast Yann LeCun
  • 78. Learning Learning Mid-Level Features Mid-Level Features [Boureau et al., CVPR 2010, ICML 2010, CVPR 2011] Yann LeCun
  • 79. ConvNets and “Conventional” Vision Architectures are Similar ConvNets and “Conventional” Vision Architectures are Similar Filter Non­ feature Filter Non­ feature Classifier Bank  Linearity Pooling  Bank  Linearity Pooling  Oriented Histogram K­means Pyramid SVM with WTA  Edges (sum) Histogram Histogram (sum) Intersection SIFT kernel Can't we use the same tricks as ConvNets to train the second stage of a “conventional vision architecture? Stage 1: SIFT Stage 2: sparse coding over neighborhoods + pooling Yann LeCun
  • 80. Using DL/ConvNet ideas in “conventional” recognition Using DL/ConvNet ideas in “conventional” recognition systems systems [Boureau et al. CVPR 2010] Adapting insights from ConvNets: Jointly encoding spatial neighborhoods instead of single points: increase spatial receptive fields for higher-level features Use max pooling instead of average pooling Train supervised dictionary for sparse coding This yields state-of-the-art results: 75.7% on Caltech-101 (+/-1.1%): record for single system 85.6% on 15-Scenes (+/- 0.2): record! Yann LeCun
  • 81. The Competition: SIFT + Sparse-Coding + PMK-SVM The Competition: SIFT + Sparse-Coding + PMK-SVM Replacing K-means with Sparse Coding [Yang 2008] [Boureau, Bach, Ponce, LeCun 2010] Yann LeCun
  • 82. Sparse Coding within Clusters Sparse Coding within Clusters Splitting the Sparse Coding into Clusters only similar things get pooled together [Boureau, et al. CVPR 2011] Yann LeCun
  • 83. Learning Learning Invariant Features Invariant Features (learning complex cells) (learning complex cells) [Kavukcuoglu, Ranzato, Fergus, LeCun, CVPR 2009] [Gregor & LeCun 2010] Yann LeCun
  • 84. Learning Invariant Features with L2 Group Sparsity Learning Invariant Features with L2 Group Sparsity Unsupervised PSD ignores the spatial pooling step. Could we devise a similar method that learns the pooling layer as well? Idea [Hyvarinen & Hoyer 2001]: group sparsity on pools of features Minimum number of pools must be non-zero Number of features that are on within a pool doesn't matter Pools tend to regroup similar features E (Y , Z )=∥Y −W d Z∥ + ∥Z − g e (W e ,Y )∥ + ∑ √∑ 2 2 2 Z k j k ∈P j i ∥Y −Y∥  2 WdZ ∑j . INPUT Y Z  ∑ k ∈ P  Z 2  j k FEATURES  i ge W e ,Y  ∥Z − Z∥  2 L2 norm within  each pool Yann LeCun
  • 85. Learning Invariant Features with L2 Group Sparsity Learning Invariant Features with L2 Group Sparsity Idea: features are pooled in group. Sparsity: sum over groups of L2 norm of activity in group. [Hyvärinen Hoyer 2001]: “subspace ICA” decoder only, square [Welling, Hinton, Osindero NIPS 2002]: pooled product of experts encoder only, overcomplete, log student-T penalty on L2 pooling [Kavukcuoglu, Ranzato, Fergus LeCun, CVPR 2010]: Invariant PSD encoder-decoder (like PSD), overcomplete, L2 pooling [Le et al. NIPS 2011]: Reconstruction ICA Same as [Kavukcuoglu 2010] with linear encoder and tied decoder [Gregor & LeCun arXiv:1006:0448, 2010] [Le et al. ICML 2012] Locally-connect non shared (tiled) encoder-decoder L2 norm within  INPUT SIMPLE  FEATURES  each pool ∑j . Encoder only (PoE, ICA), Y Decoder Only or Encoder­Decoder (iPSD, RICA) Z ∑ j 2  k ∈ P  Z k  INVARIANT FEATURES  Yann LeCun
  • 86. Pooling Similar Features using Group Sparsity Pooling Similar Features using Group Sparsity A sparse-overcomplete version of Hyvarinen's subspace ICA Decoder ensures reconstruction (unlike ICA which requires orthonogonal matrix) 1. Apply filters on a patch (with suitable non-linearity) 2. Arrange filter outputs on a 2D plane 3. square filter outputs 4. minimize sqrt of sum of blocks of squared filter outputs [Kavukcuoglu, Ranzato, Fergu, LeCun, CVPR 2009] [Jenatton, Obozinski, Bach AISTATS 2010] [Le et al. NIPS2011] Yann LeCun
  • 87. Groups are local in aa2D Topographic Map Groups are local in 2D Topographic Map The filters arrange themselves spontaneously so that similar filters enter the same pool. The pooling units can be seen as complex cells Outputs of pooling units are invariant to local transformations of the input For some it's translations, for others rotations, or other transformations. Yann LeCun
  • 88. Image-level training, local filters but no weight sharing Image-level training, local filters but no weight sharing Training on 115x115 images. Kernels are 15x15 (not shared across space!) [Gregor & LeCun 2010] Decoder Local receptive fields Reconstructed Input No shared weights 4x overcomplete L2 pooling Group sparsity over pools (Inferred) Code Predicted Code Input Encoder Yann LeCun
  • 89. Image-level training, local filters but no weight sharing Image-level training, local filters but no weight sharing Topographic maps of continuously-varying features Local overlapping pools are invariant complex cells [Gregor & LeCun arXiv:1006.0448] (double tanh encoder) [Le et al. ICML'12] (linear encoder) Yann LeCun
  • 90. Image-level training, local filters but no weight sharing Image-level training, local filters but no weight sharing Training on 115x115 images. Kernels are 15x15 (not shared across space!) Yann LeCun
  • 91. K Obermayer and GG Blasdel, Journal of  Neuroscience, Vol 13, 4114­4129 (Monkey) 119x119 Image Input 100x100 Code 20x20 Receptive field size sigma=5 Michael C. Crair, et. al. The Journal of Neurophysiology  Vol. 77 No. 6 June 1997, pp. 3381­3385 (Cat)
  • 92. Image-level training, local filters but no weight sharing Image-level training, local filters but no weight sharing Color indicates orientation (by fitting Gabors) Yann LeCun
  • 93. Theory of Repeated [Filter Bank → L2 Pooling → Average Pooling] Theory of Repeated [Filter Bank → L2 Pooling → Average Pooling] Stéphane Mallat's “Scattering Transform”: Theory of ConvNet-like architectures [Mallat & Bruna CVPR 2011] Classification with Scattering Operators [Mallat & Bruna, arXiv:1203.1513 2012] Invariant Scattering Convolution Networks [Mallat CPAM 2012] Group Invariant Scattering Yann LeCun
  • 94. Sparse Coding Using Sparse Coding Using Lateral Inhibition Lateral Inhibition [Gregor, Szlam, LeCun, NIPS 2011] Yann LeCun
  • 95. Invariant Features Lateral Inhibition Invariant Features Lateral Inhibition Replace the L1 sparsity term by a lateral inhibition matrix Easy way to impose some structure on the sparsity [Gregor, Szlam, LeCun NIPS 2011] Yann LeCun
  • 96. Invariant Features via Lateral Inhibition: Structured Sparsity Invariant Features via Lateral Inhibition: Structured Sparsity Each edge in the tree indicates a zero in the S matrix (no mutual inhibition) Sij is larger if two neurons are far away in the tree Yann LeCun
  • 100. Invariant Features via Lateral Inhibition: Topographic Maps Invariant Features via Lateral Inhibition: Topographic Maps Non-zero values in S form a ring in a 2D topology Input patches are high-pass filtered Yann LeCun
  • 101. Invariant Features via Lateral Inhibition: Topographic Maps Invariant Features via Lateral Inhibition: Topographic Maps Non-zero values in S form a ring in a 2D topology Left: no high-pass filtering of input Right: patch-level mean removal Yann LeCun
  • 102. Invariant Features via Lateral Excitation: Topographic Maps Invariant Features via Lateral Excitation: Topographic Maps Short-range lateral excitation + L1 sparsity Yann LeCun
  • 103. Learning What/Where Learning What/Where Features with Features with Temporal Constancy Temporal Constancy [Gregor & LeCun arXiv:1006.0448, 2010] Yann LeCun
  • 104. Invariant Features through Temporal Constancy Invariant Features through Temporal Constancy Object is cross-product of object type and instantiation parameters Mapping units [Hinton 1981], capsules [Hinton 2011] small medium large Object type [Karol Gregor et al.] Object size Yann LeCun
  • 105. What-Where Auto-Encoder Architecture What-Where Auto-Encoder Architecture Decoder Predicted St St­1 St­2 input W1 W1 W1 W2 t t­1 t­2 t Inferred  C 1 C 1 C 1 C 2 code C t C t­1 C t­2 C t Predicted 1 1 1 2 code f 1 1 f °W1 2 f °W f °W W  W2 W2 t t­1 t­2 Yann LeCun Encoder S S S Input
  • 106. Low-Level Filters Connected to Each Complex Cell Low-Level Filters Connected to Each Complex Cell C1 (where) C2 (what) Yann LeCun
  • 107. Generating from the Network Generating from the Network Input Yann LeCun
  • 108. Hardware Hardware Implementations Implementations Yann LeCun
  • 109. Higher End: FPGA with NeuFlow architecture Higher End: FPGA with NeuFlow architecture Now Running on Picocomputing 8x10cm high-performance FPGA board Virtex 6 LX240T: 680 MAC units, 20 neuflow tiles Full scene labeling at 20 frames/sec (50ms/frame) at 320x240 New board with Virtex­6 Yann LeCun
  • 110. NewFlow: Architecture NewFlow: Architecture MEM Multi­port memory  DMA controller (DMA) [x12 on a V6 LX240T] RISC CPU, to  CPU reconfigure tiles and  data paths, at runtime global network­on­chip to  allow fast reconfiguration grid of passive processing tiles (PTs) [x20 on a Virtex6 LX240T] Yann LeCun
  • 111. NewFlow: Processing Tile Architecture NewFlow: Processing Tile Architecture configurable router,  Term­by:­term  to stream data in  streaming  operators  and out of the tile, to  (MUL,DIV,ADD,SU neighbors or DMA  B,MAX) ports [x20] [x8,2 per tile] configurable piece­wise  linear  or quadratic  mapper [x4] configurable bank of  full 1/2D  parallel convolver  FIFOs , for stream  with 100 MAC units buffering, up to 10kB  per PT [Virtex6 LX240T] [x4] [x8] Yann LeCun
  • 112. NewFlow ASIC: 2.5x5 mm, 45nm, 0.6Watts, 160GOPS NewFlow ASIC: 2.5x5 mm, 45nm, 0.6Watts, 160GOPS Collaboration NYU-Purdue (Eugenio Culurciello's group) Suitable for vision-enabled embedded and mobile devices Status: waiting for first samples end of 2012 Yann LeCun Pham, Jelaca, Farabet, Martini, LeCun, Culurciello 2012] 
  • 113. NewFlow: Performance NewFlow: Performance Intel neuFlow neuFlow nVidia  neuFlow nVidia I7 4 cores Virtex4 Virtex 6 GT335m IBM 45nm GTX480 Peak GOP/sec 40 40 160 182 160 1350 Actual GOP/sec 12 37 147 54 147 294 FPS 14 46 182 67 182 374 Power  (W) 50 10 10 30 0.6 220 Embed? (GOP/s/W) 0.24 3.7 14.7 1.8 245 1.34 Yann LeCun
  • 114. Software Platforms: Software Platforms: Torch7 Torch7 http://www.torch.ch http://www.torch.ch [Collobert, Kavukcuoglu, Farabet 2011] Yann LeCun
  • 115. Torch7 Torch7 ML/Vision development environment Collaboration between IDIAP, NYU, and NEC Labs Uses the Lua interpreter/compiler as a front-end Very simple and efficient scripting language Ultra simple and natural syntax It's like Scheme underneath, but with a “normal” syntax. Lua is widely used in the video game and web industries Torch7 extend Lua with Numerical, ML, Vision libraries Powerful Vector/Matrix/Tensor Engine (developed by us) Interfaces to numerical libraries, OpenCV, etc Back-ends for SSE, OpenMP, CUDA, ARM-Neon,... Qt-based GUI and graphics Open source: http://www.torch.ch/ First released in early 2012 Yann LeCun
  • 116. MATRIX and VECTOR Operations Torch7 Torch7 > M = torch.Tensor(2,2):fill(2) > N = torch.Tensor(2,4):fill(3) ML/Vision development environment > x = torch.Tensor(2):fill(4) As simple as Matlab, but faster and > y = torch.Tensor(2):fill(5) better designed as a language. > = x*y -- dot product 40 2D CONVOLUTION > = M*x --- matrix-vector x = torch.rand(100,100) k = torch.rand(10,10) 16 res1 = torch.conv2(x,k) 16 [torch.Tensor of dimension 2] TRAINING A NEURAL NET mlp = nn.Sequential() mlp:add( nn.Linear(10, 25) ) -- 10 input, 25 hidden units mlp:add( nn.Tanh() ) -- some hyperbolic tangent transfer function mlp:add( nn.Linear(25, 1) ) -- 1 output criterion = nn.MSECriterion() -- Mean Squared Error criterion trainer = nn.StochasticGradient(mlp, criterion) trainer:train(dataset) -- train using some examples Yann LeCun
  • 117. Acknowledgements Acknowledgements Camille Couprie Karol Gregor Y-Lan Boureau Clément Farabet Kevin Jarrett Arthur Szlam Koray Kavukcuoglu Marc'Aurelio Ranzato Rob Fergus Pierre Sermanet Laurent Najman (ESIEE) Eugenio Culurciello (Purdue) Yann LeCun
  • 118. The End The End Yann LeCun