![Deep Learning-based Image Compression
Auto Encoder
�
𝑦𝑦 = 𝑄𝑄(𝑦𝑦)
𝑦𝑦 = 𝑔𝑔𝑎𝑎 𝑥𝑥; 𝜙𝜙𝑔𝑔
�
𝑥𝑥 = 𝑔𝑔𝑠𝑠(�
𝑦𝑦; 𝜃𝜃𝑔𝑔)
𝑅𝑅 = 𝐸𝐸𝑥𝑥~𝑝𝑝𝑥𝑥
[− log2 𝑝𝑝�
𝑦𝑦(�
𝑦𝑦)]
𝐷𝐷 = 𝑥𝑥 − �
𝑥𝑥 2
2
???](https://image.slidesharecdn.com/pr395v2-220718072451-96b86d40/85/PR-395-Variational-Image-Compression-with-a-Scale-Hyperprior-15-320.jpg)
![Problem of Quantization
Adding Uniform Noise [PR328]
�
𝑦𝑦 = 𝑄𝑄 𝑦𝑦 �
𝑦𝑦 = 𝑦𝑦 + 𝑛𝑛, 𝑛𝑛~𝑈𝑈(−
1
2
,
1
2
)
Ballé, Johannes, Valero Laparra, and Eero P. Simoncelli. "End-to-end optimized image compression." 5th International Conference on Learning Representations, ICLR 2017.
𝑅𝑅 = 𝐸𝐸𝑥𝑥~𝑝𝑝𝑥𝑥
− log2 𝑝𝑝�
𝑦𝑦 �
𝑦𝑦 = 𝐸𝐸𝑥𝑥~𝑝𝑝𝑥𝑥
[− log2 𝑝𝑝�
𝑦𝑦(𝑄𝑄(𝑔𝑔𝑎𝑎(𝑥𝑥; 𝜙𝜙𝑔𝑔)))]
𝑅𝑅 = 𝐸𝐸𝑥𝑥~𝑝𝑝𝑥𝑥
− log2 𝑝𝑝�
𝑦𝑦 �
𝑦𝑦 = 𝐸𝐸𝑥𝑥~𝑝𝑝𝑥𝑥
[− log2 𝑝𝑝�
𝑦𝑦(𝑔𝑔𝑎𝑎 𝑥𝑥; 𝜙𝜙𝑔𝑔 + 𝑛𝑛)]](https://image.slidesharecdn.com/pr395v2-220718072451-96b86d40/85/PR-395-Variational-Image-Compression-with-a-Scale-Hyperprior-17-320.jpg)
![Rate Loss
Applying Uniform Noise Approximation
𝑅𝑅 = 𝐸𝐸𝑥𝑥~𝑝𝑝𝑥𝑥
[− log2 𝑝𝑝�
𝑦𝑦(𝑔𝑔𝑎𝑎 𝑥𝑥; 𝜙𝜙𝑔𝑔 + 𝑛𝑛)]
𝑝𝑝�
𝑦𝑦
𝑅𝑅 = 𝐸𝐸𝑥𝑥~𝑝𝑝𝑥𝑥
[− log2 𝑝𝑝�
𝑦𝑦(�
𝑦𝑦)]
Entropy Model](https://image.slidesharecdn.com/pr395v2-220718072451-96b86d40/85/PR-395-Variational-Image-Compression-with-a-Scale-Hyperprior-19-320.jpg)
![Non-parametric Density Model
Revisiting Loss Function
𝑅𝑅 = 𝐸𝐸𝑥𝑥~𝑝𝑝𝑥𝑥
[− log2 𝑝𝑝�
𝑦𝑦(�
𝑦𝑦)]
𝐷𝐷 = 𝑥𝑥 − �
𝑥𝑥 2
2
𝐿𝐿 = 𝑅𝑅 + 𝜆𝜆𝜆𝜆](https://image.slidesharecdn.com/pr395v2-220718072451-96b86d40/85/PR-395-Variational-Image-Compression-with-a-Scale-Hyperprior-26-320.jpg)
![Scale Hyperprior
Factorized Prior
�
𝑦𝑦 = 𝑦𝑦 + 𝑛𝑛
𝑦𝑦 = 𝑔𝑔𝑎𝑎 𝑥𝑥; 𝜙𝜙𝑔𝑔
�
𝑥𝑥 = 𝑔𝑔𝑠𝑠(�
𝑦𝑦; 𝜃𝜃𝑔𝑔)
𝑅𝑅 = − log2 𝑝𝑝�
𝑦𝑦(�
𝑦𝑦)
𝐷𝐷 = 𝑥𝑥 − �
𝑥𝑥 2
2
𝐿𝐿 = 𝑅𝑅 + 𝜆𝜆𝜆𝜆
Inference Model
Generative Model
Entropy Model
𝐸𝐸𝐸𝐸𝐸𝐸[�
𝑦𝑦]
𝐷𝐷𝐷𝐷𝐷𝐷[�
𝑦𝑦]](https://image.slidesharecdn.com/pr395v2-220718072451-96b86d40/85/PR-395-Variational-Image-Compression-with-a-Scale-Hyperprior-34-320.jpg)
![Scale Hyperprior
Factorized Prior
�
𝑦𝑦 = 𝑦𝑦 + 𝑛𝑛
𝑦𝑦 = 𝑔𝑔𝑎𝑎 𝑥𝑥; 𝜙𝜙𝑔𝑔
�
𝑥𝑥 = 𝑔𝑔𝑠𝑠(�
𝑦𝑦; 𝜃𝜃𝑔𝑔)
𝑅𝑅 = − log2 𝑝𝑝�
𝑦𝑦 �
𝑦𝑦 − log2 𝑝𝑝�
𝑦𝑦|�
𝑧𝑧(�
𝑦𝑦| ̃
𝑧𝑧)
𝐷𝐷 = 𝑥𝑥 − �
𝑥𝑥 2
2
𝐿𝐿 = 𝑅𝑅 + 𝜆𝜆𝜆𝜆
Inference Model
Generative Model
Entropy Model
𝑧𝑧 = ℎ𝑎𝑎 𝑦𝑦; 𝜙𝜙ℎ
̃
𝑧𝑧 = 𝑧𝑧 + 𝑛𝑛
�
𝜎𝜎 = ℎ𝑠𝑠 ̃
𝑧𝑧; 𝜃𝜃ℎ 𝐸𝐸𝐸𝐸𝐸𝐸[�
𝑦𝑦; �
𝜎𝜎]
𝐷𝐷𝐷𝐷𝐷𝐷[ ̃
𝑧𝑧] �
𝜎𝜎 = ℎ𝑠𝑠 ̃
𝑧𝑧; 𝜃𝜃ℎ
𝐸𝐸𝐸𝐸𝐸𝐸[ ̃
𝑧𝑧]
𝐷𝐷𝐷𝐷𝐷𝐷[�
𝑦𝑦; �
𝜎𝜎]](https://image.slidesharecdn.com/pr395v2-220718072451-96b86d40/85/PR-395-Variational-Image-Compression-with-a-Scale-Hyperprior-37-320.jpg)
PR-395: Variational Image Compression with a Scale Hyperprior
- 1. Variational Image Compression with a Scale Hyperprior Hyeongmin Lee PR-395 2022.7.17
- 3. Entropy Coding Image Compression 0011010100111...
- 4. Entropy Coding Entropy Coding A,B,C,D: 4 Letters (need at least 2 bits per letter) Sample Bits A 00 B 10 C 01 D 11 100100001111110101000100010001 (30 bits) https://www.programiz.com/dsa/huffman-coding
- 5. Entropy Coding Entropy Coding A,B,C,D: 4 Letters (need at least 2 bits per letter) Sample Bits A 11 B 100 C 0 D 101 1000111110110110100110110110 (28 bits) Lower Bound?? Huffman Coding
- 6. Entropy Coding Entropy 𝑯𝑯 = − � 𝒑𝒑𝒊𝒊𝒍𝒍𝒍𝒍𝒈𝒈𝟐𝟐𝒑𝒑𝒊𝒊 Sample 𝒑𝒑𝒊𝒊 A 5/15 B 1/15 C 6/15 D 3/15
- 8. Image Compression Image Compression 0011010100111... Lossless Coding: bmp Coding the Integers Itself Lossy Coding: JPEG Reducing the ‘Entropy’ of samples
- 9. Image Compression JPEG 8x8 cutting Discrete Cosine Transform Quantization Entropy Coding... https://www.whydomath.org/node/wavlets/basicjpg.html
- 10. Image Compression JPEG High Quality High Entropy High Bitrate Low Quality Low Entropy Low Bitrate
- 11. Image Compression Bitrate-Distortion Tradeoff
- 13. Deep Learning-based Image Compression Image Compression Pipeline Transform Quantization Entropy Coding Decoding Inverse Transform
- 14. Deep Learning-based Image Compression Image Compression Pipeline 𝐿𝐿 = 𝑅𝑅 + 𝜆𝜆𝜆𝜆
- 15. Deep Learning-based Image Compression Auto Encoder � 𝑦𝑦 = 𝑄𝑄(𝑦𝑦) 𝑦𝑦 = 𝑔𝑔𝑎𝑎 𝑥𝑥; 𝜙𝜙𝑔𝑔 � 𝑥𝑥 = 𝑔𝑔𝑠𝑠(� 𝑦𝑦; 𝜃𝜃𝑔𝑔) 𝑅𝑅 = 𝐸𝐸𝑥𝑥~𝑝𝑝𝑥𝑥 [− log2 𝑝𝑝� 𝑦𝑦(� 𝑦𝑦)] 𝐷𝐷 = 𝑥𝑥 − � 𝑥𝑥 2 2 ???
- 17. Problem of Quantization Adding Uniform Noise [PR328] � 𝑦𝑦 = 𝑄𝑄 𝑦𝑦 � 𝑦𝑦 = 𝑦𝑦 + 𝑛𝑛, 𝑛𝑛~𝑈𝑈(− 1 2 , 1 2 ) Ballé, Johannes, Valero Laparra, and Eero P. Simoncelli. "End-to-end optimized image compression." 5th International Conference on Learning Representations, ICLR 2017. 𝑅𝑅 = 𝐸𝐸𝑥𝑥~𝑝𝑝𝑥𝑥 − log2 𝑝𝑝� 𝑦𝑦 � 𝑦𝑦 = 𝐸𝐸𝑥𝑥~𝑝𝑝𝑥𝑥 [− log2 𝑝𝑝� 𝑦𝑦(𝑄𝑄(𝑔𝑔𝑎𝑎(𝑥𝑥; 𝜙𝜙𝑔𝑔)))] 𝑅𝑅 = 𝐸𝐸𝑥𝑥~𝑝𝑝𝑥𝑥 − log2 𝑝𝑝� 𝑦𝑦 � 𝑦𝑦 = 𝐸𝐸𝑥𝑥~𝑝𝑝𝑥𝑥 [− log2 𝑝𝑝� 𝑦𝑦(𝑔𝑔𝑎𝑎 𝑥𝑥; 𝜙𝜙𝑔𝑔 + 𝑛𝑛)]
- 18. Rate Loss
- 19. Rate Loss Applying Uniform Noise Approximation 𝑅𝑅 = 𝐸𝐸𝑥𝑥~𝑝𝑝𝑥𝑥 [− log2 𝑝𝑝� 𝑦𝑦(𝑔𝑔𝑎𝑎 𝑥𝑥; 𝜙𝜙𝑔𝑔 + 𝑛𝑛)] 𝑝𝑝� 𝑦𝑦 𝑅𝑅 = 𝐸𝐸𝑥𝑥~𝑝𝑝𝑥𝑥 [− log2 𝑝𝑝� 𝑦𝑦(� 𝑦𝑦)] Entropy Model
- 21. Non-parametric Density Model Defining the density as a function (Neural Network)
- 22. Non-parametric Density Model Using Sigmoid at the last layer All Jacobian elements are positive
- 23. Non-parametric Density Model Intermediate Activation Functions All Jacobian elements are positive 𝑎𝑎 > 0 −1 < 𝑎𝑎 < 0 𝑎𝑎 < −1
- 24. Non-parametric Density Model Setting parameter constraints All Jacobian elements are positive
- 25. Non-parametric Density Model Experiment on toy example
- 26. Non-parametric Density Model Revisiting Loss Function 𝑅𝑅 = 𝐸𝐸𝑥𝑥~𝑝𝑝𝑥𝑥 [− log2 𝑝𝑝� 𝑦𝑦(� 𝑦𝑦)] 𝐷𝐷 = 𝑥𝑥 − � 𝑥𝑥 2 2 𝐿𝐿 = 𝑅𝑅 + 𝜆𝜆𝜆𝜆
- 28. Variational Auto Encoder Auto Encoder � 𝑦𝑦 = 𝑦𝑦 + 𝑛𝑛 𝑦𝑦 = 𝑔𝑔𝑎𝑎 𝑥𝑥; 𝜙𝜙𝑔𝑔 � 𝑥𝑥 = 𝑔𝑔𝑠𝑠(� 𝑦𝑦; 𝜃𝜃𝑔𝑔)
- 29. Variational Auto Encoder Variational Auto Encoder � 𝑦𝑦~𝑝𝑝� 𝑦𝑦|𝑥𝑥(� 𝑦𝑦|𝑥𝑥) 𝑥𝑥~𝑝𝑝𝑥𝑥|𝑦𝑦(𝑥𝑥|� 𝑦𝑦)
- 30. Variational Auto Encoder Defining Generative Model � 𝑦𝑦~𝑝𝑝� 𝑦𝑦|𝑥𝑥(� 𝑦𝑦|𝑥𝑥)
- 31. Variational Auto Encoder Setting the parametric version of posterior
- 32. Variational Auto Encoder Setting KL Divergence
- 33. Scale Hyperprior
- 34. Scale Hyperprior Factorized Prior � 𝑦𝑦 = 𝑦𝑦 + 𝑛𝑛 𝑦𝑦 = 𝑔𝑔𝑎𝑎 𝑥𝑥; 𝜙𝜙𝑔𝑔 � 𝑥𝑥 = 𝑔𝑔𝑠𝑠(� 𝑦𝑦; 𝜃𝜃𝑔𝑔) 𝑅𝑅 = − log2 𝑝𝑝� 𝑦𝑦(� 𝑦𝑦) 𝐷𝐷 = 𝑥𝑥 − � 𝑥𝑥 2 2 𝐿𝐿 = 𝑅𝑅 + 𝜆𝜆𝜆𝜆 Inference Model Generative Model Entropy Model 𝐸𝐸𝐸𝐸𝐸𝐸[� 𝑦𝑦] 𝐷𝐷𝐷𝐷𝐷𝐷[� 𝑦𝑦]
- 35. Scale Hyperprior Scale of Y Encoding Scale as ‘Additional Information’
- 36. Scale Hyperprior Scale Hyperprior
- 37. Scale Hyperprior Factorized Prior � 𝑦𝑦 = 𝑦𝑦 + 𝑛𝑛 𝑦𝑦 = 𝑔𝑔𝑎𝑎 𝑥𝑥; 𝜙𝜙𝑔𝑔 � 𝑥𝑥 = 𝑔𝑔𝑠𝑠(� 𝑦𝑦; 𝜃𝜃𝑔𝑔) 𝑅𝑅 = − log2 𝑝𝑝� 𝑦𝑦 � 𝑦𝑦 − log2 𝑝𝑝� 𝑦𝑦|� 𝑧𝑧(� 𝑦𝑦| ̃ 𝑧𝑧) 𝐷𝐷 = 𝑥𝑥 − � 𝑥𝑥 2 2 𝐿𝐿 = 𝑅𝑅 + 𝜆𝜆𝜆𝜆 Inference Model Generative Model Entropy Model 𝑧𝑧 = ℎ𝑎𝑎 𝑦𝑦; 𝜙𝜙ℎ ̃ 𝑧𝑧 = 𝑧𝑧 + 𝑛𝑛 � 𝜎𝜎 = ℎ𝑠𝑠 ̃ 𝑧𝑧; 𝜃𝜃ℎ 𝐸𝐸𝐸𝐸𝐸𝐸[� 𝑦𝑦; � 𝜎𝜎] 𝐷𝐷𝐷𝐷𝐷𝐷[ ̃ 𝑧𝑧] � 𝜎𝜎 = ℎ𝑠𝑠 ̃ 𝑧𝑧; 𝜃𝜃ℎ 𝐸𝐸𝐸𝐸𝐸𝐸[ ̃ 𝑧𝑧] 𝐷𝐷𝐷𝐷𝐷𝐷[� 𝑦𝑦; � 𝜎𝜎]
- 38. Scale Hyperprior Network Architecture
- 41. Thank You!