Semantic Segmentation - Fully Convolutional Networks for Semantic Segmentation
1 like1,670 views
This document discusses several semantic segmentation methods using deep learning, including fully convolutional networks (FCNs), U-Net, and SegNet. FCNs were among the first to use convolutional networks for dense, pixel-wise prediction by converting classification networks to fully convolutional form and combining coarse and fine feature maps. U-Net and SegNet are encoder-decoder architectures that extract high-level semantic features from the input image and then generate pixel-wise predictions, with U-Net copying and cropping features and SegNet using pooling indices for upsampling. These methods demonstrate that convolutional networks can effectively perform semantic segmentation through dense prediction.
![DeconvolutionDeconvolution
Deconvolutional network [2015]](https://image.slidesharecdn.com/semanticsegmentationslides-181218144057/85/Semantic-Segmentation-Fully-Convolutional-Networks-for-Semantic-Segmentation-13-320.jpg)
![FCN-8s SDS [17] Ground Truth Image](https://image.slidesharecdn.com/semanticsegmentationslides-181218144057/85/Semantic-Segmentation-Fully-Convolutional-Networks-for-Semantic-Segmentation-20-320.jpg)
![Deconvolutional network [2015]Deconvolutional network [2015]
Learning Deconvolution Network for Semantic Segmentation
(https://arxiv.org/abs/1505.04366)](https://image.slidesharecdn.com/semanticsegmentationslides-181218144057/85/Semantic-Segmentation-Fully-Convolutional-Networks-for-Semantic-Segmentation-23-320.jpg)
![Deconvolutional networkDeconvolutional network
[1]](https://image.slidesharecdn.com/semanticsegmentationslides-181218144057/85/Semantic-Segmentation-Fully-Convolutional-Networks-for-Semantic-Segmentation-24-320.jpg)
![U-Net [2015]U-Net [2015]
copy and crop
input
image
tile
output
segmentation
map
641
128
256
512
1024
max pool 2x2
up-conv 2x2
conv 3x3, ReLU
572x572
284²
64
128
256
512
570x570
568x568
282²
280²140²
138²
136²68²
66²
64²32²
28²
56²
54²
52²
512
104²
102²
100²200²
30²
198²
196²392x392
390x390
388x388
388x388
1024
512 256
256 128
64128 64 2
conv 1x1](https://image.slidesharecdn.com/semanticsegmentationslides-181218144057/85/Semantic-Segmentation-Fully-Convolutional-Networks-for-Semantic-Segmentation-25-320.jpg)
![SegNet [2015, University of Cambridge]SegNet [2015, University of Cambridge]
Convolutional Encoder-Decoder Architecture
Convolutional Encoder-Decoder
Pooling Indices
Input
Segmentation
Output
Conv + Batch Normalisation + ReLU
Pooling Upsampling Softmax
RGB Image](https://image.slidesharecdn.com/semanticsegmentationslides-181218144057/85/Semantic-Segmentation-Fully-Convolutional-Networks-for-Semantic-Segmentation-28-320.jpg)
![Q & AQ & A
ReferenceReference
[1]
[2]
[3]
A brief introduction to recent segmentation methods
(https://www.slideshare.net/mitmul/a-brief-introduction-to-recent-
segmentation-methods)
关于FCN 论⽂中的Shift-and-stitch 的详尽解释
(https://www.jianshu.com/p/e534e2be5d7d)
A 2017 Guide to Semantic Segmentation with Deep Learning
(http://blog.qure.ai/notes/semantic-segmentation-deep-learning-review)](https://image.slidesharecdn.com/semanticsegmentationslides-181218144057/85/Semantic-Segmentation-Fully-Convolutional-Networks-for-Semantic-Segmentation-31-320.jpg)
Semantic Segmentation - Fully Convolutional Networks for Semantic Segmentation
- 1. Semantic SegmentationSemantic Segmentation Example - Fully Convolutional Networks for SemanticExample - Fully Convolutional Networks for Semantic SegmentationSegmentation UC Berkeley
- 2. Computer visionComputer vision picture source (https://read01.com/Bng557M.html#.W4T_kXUzbiw)
- 3. Semantic segmentationSemantic segmentation Each pixel has its own label! picture source (https://www.quora.com/What-does-the-term- semantic-segmentation-mean-in-the-context-of-Deep-Learning)
- 4. Typical wayTypical way Image Model Outcome w x h Label w x h cross entropy Loss is calculated for each pixel independently.
- 5. IssueIssue How to create dense prediction? related works: patchwise training small model -> small receptive eld post-processing (e.g. superpixel projection, random eld regularization, ltering ...) saturating tanh restricted receptive eld input shifting and output interlacing multi-scale pyramid processing
- 7. IdeaIdea Semantics and location Global information resolves what while local information resolves where. global information -> what (semantics) local information -> where (location)
- 8. IdeaIdea Use train by entire image, instead of patch. Let receptive eld overlap signi cantly to improve ef ciency. Transfer learning from classi cation net to fully convolution network. For pixelwise prediction, connect coarse outputs to pixels.
- 9. Fully convolutional networkFully convolutional network Fully Convolutional Networks for Semantic Segmentation (https://people.eecs.berkeley.edu/~jonlong/long_shelhamer_fcn.pdf)
- 10. Convert classi cation net to fully convolution networkConvert classi cation net to fully convolution network
- 11. Dense predictionDense prediction Strategy for upsampling: Shift-and-stitch Deconvolution
- 14. Evaluation methodEvaluation method is the number of the pixel of class predicted to be class there are different classes , total number of pixels of class pixel accuracy: mean accuracy: mean region intersection over union (IU): nij i j ncl =ti ∑ j nij i /∑ i nii ∑ i ti (1/ ) /ncl ∑ i nii ∑ i ti 1 ncl ∑ i nii + −ti ∑ j nji nii
- 15. ResultsResults
- 16. Combine coarse and ne feature mapsCombine coarse and ne feature maps
- 17. Combine coarse and ne feature mapsCombine coarse and ne feature maps
- 18. Combine coarse and ne feature mapsCombine coarse and ne feature maps 32x upsampling FCN-32sconv 7 conv 7 pool 4 2x upsampling 1 x 1 conv + 16x upsampling FCN-16s pool 3 2x upsampling conv 7 pool 4 4x upsampling 1 x 1 conv + 8x upsampling FCN-8s
- 19. ResultsResults
- 20. FCN-8s SDS [17] Ground Truth Image
- 21. ImportanceImportance FCN for pixelwise prediction arbitrary-sized inputs learning and inference whole image at a time leverage supervised pre-train model upsampling (deconvolution)
- 22. Take home messageTake home message more convolution, more coarse combine coarse and ne feature map (skip architecture)
- 23. Deconvolutional network [2015]Deconvolutional network [2015] Learning Deconvolution Network for Semantic Segmentation (https://arxiv.org/abs/1505.04366)
- 25. U-Net [2015]U-Net [2015] copy and crop input image tile output segmentation map 641 128 256 512 1024 max pool 2x2 up-conv 2x2 conv 3x3, ReLU 572x572 284² 64 128 256 512 570x570 568x568 282² 280²140² 138² 136²68² 66² 64²32² 28² 56² 54² 52² 512 104² 102² 100²200² 30² 198² 196²392x392 390x390 388x388 388x388 1024 512 256 256 128 64128 64 2 conv 1x1
- 26. U-Net: Convolutional Networks for Biomedical Image Segmentation (https://arxiv.org/abs/1505.04597)
- 27. U-NetU-Net U-Net: Convolutional Networks for Biomedical Image Segmentation (https://arxiv.org/abs/1505.04597)
- 28. SegNet [2015, University of Cambridge]SegNet [2015, University of Cambridge] Convolutional Encoder-Decoder Architecture Convolutional Encoder-Decoder Pooling Indices Input Segmentation Output Conv + Batch Normalisation + ReLU Pooling Upsampling Softmax RGB Image
- 29. SegNetSegNet high ef ciency reduce parameters make end-to-end training availible
- 30. My conclusionMy conclusion Encoder-decoder architecture Encoder: extract high-level or abstract meanings (semantics) Decoder: generate instance from abstract meanings Discriminative model Generative model P (y ∣ x) P (x, y)
- 31. Q & AQ & A ReferenceReference [1] [2] [3] A brief introduction to recent segmentation methods (https://www.slideshare.net/mitmul/a-brief-introduction-to-recent- segmentation-methods) 关于FCN 论⽂中的Shift-and-stitch 的详尽解释 (https://www.jianshu.com/p/e534e2be5d7d) A 2017 Guide to Semantic Segmentation with Deep Learning (http://blog.qure.ai/notes/semantic-segmentation-deep-learning-review)