Using neon for pattern recognition in audio data
1 like784 views
This document provides an outline and overview of a recurrent neural network meetup focused on using the neon deep learning framework. The meetup covers an introduction to neon, setting up the workshop environment, convolutional neural network theory and hands-on examples, recurrent neural network theory and hands-on examples including for audio data processing. Benchmark results are shown indicating neon's performance. Examples demonstrated include music genre classification using CNNs and whale call identification using RNNs.
Using neon for pattern recognition in audio data
- 1. Anil Thomas Recurrent Neural Hacks Meetup July 16, 2016 MAKING MACHINES SMARTER.™ Using neon for pattern recognition in audio data
- 2. Outline 2 • Intro to neon • Workshop environment setup • CNN theory • CNN hands-on • RNN theory • RNN hands-on
- 3. NEON 3
- 4. Neon 4 Backends NervanaCPU, NervanaGPU NervanaMGPU, NervanaEngine (internal) Datasets Images: ImageNet, CIFAR-10, MNIST Captions: flickr8k, flickr30k, COCO; Text: Penn Treebank, hutter-prize, IMDB, Amazon Initializers Constant, Uniform, Gaussian, Glorot Uniform Learning rules Gradient Descent with Momentum RMSProp, AdaDelta, Adam, Adagrad Activations Rectified Linear, Softmax, Tanh, Logistic Layers Linear, Convolution, Pooling, Deconvolution, Dropout Recurrent, Long Short-Term Memory, Gated Recurrent Unit, Recurrent Sum, LookupTable Costs Binary Cross Entropy, Multiclass Cross Entropy, Sum of Squares Error Metrics Misclassification, TopKMisclassification, Accuracy • Modular components • Extensible, OO design • Documentation • neon.nervanasys.com
- 5. Why neon? 5 • Fastest according to third-party benchmarks • Advanced data-loading capabilities • Open source (including the optimized GPU kernels!) • Support for distributed computing
- 7. Data loading in neon 7 • Multithreaded • Non-blocking IO • Non-blocking decompression and augmentation • Supports different types of data (We will focus on audio in this workshop) • Automatic ingest • Can handle huge datasets
- 9. Github repo with source code 9 • https://github.com/anlthms/meetup2 • Music classification examples • To clone locally: git clone https://github.com/anlthms/meetup2.git
- 10. Configuring EC2 instance 10 • Use your own EC2 account at http://aws.amazon.com/ec2/ • Select US West (N. California) zone • Search for nervana-neon10 within Community AMIs • Select g2.2xlarge as instance type • After launching the instance, log in and activate virtual env: source ~/neon/.venv/bin/activate
- 11. Datasets 11 • GTZAN music genre dataset 10 genres 100 clips in each genre Each clip is 30 seconds long Preloaded in the AMI at /home/ubuntu/nervana/music/ • Whale calls dataset from Kaggle https://www.kaggle.com/c/whale-detection-challenge/data 30,000 2-second sound clips Preloaded in the AMI at /home/ubuntu/nervana/wdc/
- 12. CNN THEORY 12
- 13. Convolution 0 1 2 3 4 5 6 7 8 0 1 2 3 19 25 37 43 0 1 3 4 0 1 2 3 19 13 • Each element in the output is the result of a dot product between two vectors
- 14. Convolutional layer 14 0 1 2 3 4 5 6 7 8 19 14 0 1 2 3 4 5 6 7 8 0 1 2 3 19 25 37 43 0 2 3 1 0 2 3 1 0 2 3 1 0 2 3 1 25 37 43
- 15. Convolutional layer 15 x + x + x + x = 00 11 32 43 The weights are shared among the units. 0 1 2 3 4 5 6 7 8 0 2 3 1 0 2 3 1 0 2 3 1 0 2 3 1 19 19
- 16. Recognizing patterns 16 Detected the pattern!
- 17. 17 B0 B1 B2 B3 B4 B5 B6 B7 B8 G0 G1 G2 G3 G4 G5 G6 G7 G8 R0 R1 R2 R3 R4 R5 R6 R7 R8
- 18. 18 B0 B1 B2 B3 B4 B5 B6 B7 B8 G0 G1 G2 G3 G4 G5 G6 G7 G8 R0 R1 R2 R3 R4 R5 R6 R7 R8
- 19. B0 B1 B2 B3 B4 B5 B6 B7 B8 19 G0 G1 G2 G3 G4 G5 G6 G7 G8 R0 R1 R2 R3 R4 R5 R6 R7 R8
- 20. B0 B1 B2 B3 B4 B5 B6 B7 B8 20 G0 G1 G2 G3 G4 G5 G6 G7 G8 R0 R1 R2 R3 R4 R5 R6 R7 R8
- 21. Max pooling 0 1 2 3 4 5 6 7 8 4 5 7 8 0 1 3 4 4 21 • Each element in the output is the maximum value within the pooling window Max( )
- 22. CNN HANDS-ON 22
- 23. CNN example 23 • Source at https://github.com/anlthms/meetup2/blob/master/cnn1.py • Command line: ./cnn1.py -e 16 -w /home/ubuntu/nervana/music -r 0 -v
- 25. Sample output of conv layer #1 25
- 26. Sample output of conv layer #2 26
- 27. RNN THEORY 27
- 28. Recurrent layer 28 x h WI WR ht = g ( WI xt + WR h(t-1) )
- 29. Recurrent layer (unrolled) 29 x1 x2 x3 h1 h2 h3 WI WI WI WR WR WR ht = g ( WI xt + WR h(t-1) ) • x represents the input sequence • h is the memory (as well as the output of the recurrent layer) • h is computed from the past memory and the current input • h summarizes the sequence up to the current time steptime
- 30. Training RNNs 30 • Exploding/vanishing gradients • Initialization • Gradient clipping • Optimizers with Adaptive learning rates (e.g. Adagrad) • LSTM to capture long term dependencies
- 31. 31 x! tanh! x! +! x! output gate! forget gate! Input gate! input! LSTM
- 32. Applications of RNNs 32 • Image captioning • Speech recognition • Machine translation • Time-series analysis
- 33. RNN HANDS-ON 33
- 34. RNN example 1 34 • Source at https://github.com/anlthms/meetup2/blob/master/rnn1.py • Command line: ./rnn1.py -e 16 -w /home/ubuntu/nervana/music -r 0 -v (Does not work well! This example demonstrates challenges of training RNNs)
- 35. RNN example 2 35 • Source at https://github.com/anlthms/meetup2/blob/master/rnn2.py • Command line: ./rnn2.py -e 16 -w /home/ubuntu/nervana/music -r 0 -v (Uses Glorot init, gradient clipping, Adagrad and LSTM)
- 36. RNN example 3 36 • Source at https://github.com/anlthms/meetup2/blob/master/rnn3.py • Command line: ./rnn3.py -e 16 -w /home/ubuntu/nervana/music -r 0 -v (Uses multiple bi-rnn layers)
- 37. Final example 37 • Source at https://github.com/NervanaSystems/neon/blob/master/examples/whale_calls.py • Command line: ./whale_calls.py -e 16 -w /home/ubuntu/nervana/wdc -r 0 -s whales.pkl -v
- 38. Network structure of final example 38 Convolution Layer 'Convolution_0': 1 x (81x49) inputs, 128 x (79x24) outputs, 0,0 padding, 1,2 stride Activation Layer 'Convolution_0_Rectlin': Rectlin Convolution Layer 'Convolution_1': 128 x (79x24) inputs, 256 x (77x22) outputs, 0,0 padding, 1,1 stride BatchNorm Layer 'Convolution_1_bnorm': 433664 inputs, 1 steps, 256 feature maps Activation Layer 'Convolution_1_Rectlin': Rectlin Pooling Layer 'Pooling_0': 256 x (77x22) inputs, 256 x (38x11) outputs Convolution Layer 'Convolution_2': 256 x (38x11) inputs, 512 x (37x10) outputs, 0,0 padding, 1,1 stride BatchNorm Layer 'Convolution_2_bnorm': 189440 inputs, 1 steps, 512 feature maps Activation Layer 'Convolution_2_Rectlin': Rectlin BiRNN Layer 'BiRNN_0': 18944 inputs, (256 outputs) * 2, 10 steps BiRNN Layer 'BiRNN_1': (256 inputs) * 2, (256 outputs) * 2, 10 steps BiRNN Layer 'BiRNN_2': (256 inputs) * 2, (256 outputs) * 2, 10 steps RecurrentOutput choice RecurrentLast : (512, 10) inputs, 512 outputs Linear Layer 'Linear_0': 512 inputs, 32 outputs BatchNorm Layer 'Linear_0_bnorm': 32 inputs, 1 steps, 32 feature maps Activation Layer 'Linear_0_Rectlin': Rectlin Linear Layer 'Linear_1': 32 inputs, 2 outputs Activation Layer 'Linear_1_Softmax': Softmax
- 39. More info 39 Nervana’s deep learning tutorials: https://www.nervanasys.com/deep-learning-tutorials/ Github page: https://github.com/NervanaSystems/neon For more information, contact: [email protected]
- 40. 40 NERVANA