Intelligent Image Enhancement and Restoration - From Prior Driven Model to Advanced Deep Learning Part 2: text centric image style transfer
0 likes202 views
The document presents a tutorial on intelligent image and video editing, focusing on text-centric image style transfer and various methodological approaches to text stylization. It covers supervised and unsupervised methods for automatic text effects transfer, addressing challenges and innovations in texture generation and design. Results from user studies indicate the effectiveness of the proposed methods in achieving desirable stylistic effects in graphical designs.
Intelligent Image Enhancement and Restoration - From Prior Driven Model to Advanced Deep Learning Part 2: text centric image style transfer
- 1. IEEE ICME-2019 8:30 – 12:00, July. 08 Shanghai, China Intelligent Image/Video Editing Tutorial
- 2. STRUCT Group Jiaying Liu Wenhan Yang Text-Centric Image Style Transfer Part 2
- 3. Outline Background / 005 Statistic Based Text Stylization / 014 Context-Aware Unsupervised Text Stylization / 053 Text Effects Transfer via Stylization & Destylization / 077 Text-Centric Image Style Transfer
- 4. Outline Background / 005 Statistic Based Text Stylization / 014 Context-Aware Unsupervised Text Stylization / 053 Text Effects Transfer via Stylization & Destylization / 077 Text-Centric Image Style Transfer
- 5. STRUCT Group05 Image Style Transfer Image stylization Enrich content Intensify representation Convey emotion Application Social media Graphic design Background
- 6. STRUCT Group06 Image Style Transfer Background Taxonomy Neural Style Transfer1 CNNMRF2, Neural Doodle3 Pix2pix-cGAN, cycleGAN 42017 CVPR Image-to-Image Translation with Conditional Adversarial Networks 32016 Arxiv Semantic style transfer and turning two-bit doodles into fine artworks 12016 CVPR. Image style transfer using convolutional neural networks 22016 CVPR Combining Markov random fields and convolutional neural networks for image synthesis 52017 ICCV Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks Method Style Transfer Global based GAN Local based
- 7. STRUCT Group07 Image Style Transfer Background Taxonomy Neural Style Transfer1 Main idea: Match the global feature distributions between the style image and the target image • Gram Matrix (Coherence) • Mean & Variance • Whitening & Coloring Style as global distribution Method Style Transfer Global based GAN Local based
- 8. STRUCT Group08 Image Style Transfer Background Taxonomy CNNMRF2, Neural Doodle3 Main idea: match local patches / pixels • Patch/pixel in image domain • Image Analogy (SIGGRAPH01) • Neural patches • CNNMRF • Hybrid: Deep image analogyStyle as local patches Method Style Transfer Global based GAN Local based
- 9. STRUCT Group09 Image Style Transfer Background Taxonomy Main idea: Learn mapping between two domains Instead of defining what style is, Learn to classify the style • Paired • Pix2pix-cGAN • Unpaired • cycleGAN Pix2pix-cGAN, cycleGAN Style is learned by data Method Style Transfer Global based GAN Local based
- 10. STRUCT Group010 Background Problem Analysis Motivation Manually design: Require skills & Take too much time Automatic text effects transfer By Algorithms (seconds) By Manual work (hours) Determine color styles, Warp textures to match text shapes Adjust the transparency for visual plausibleness, etc.
- 11. STRUCT Group011 Guidance S Style S’ Target T Result T’ ① Learn transformation ② Apply transformation Image Style Transfer Supervised method Unsupervised method Background
- 12. STRUCT Group012 Extract features to build mappings Image Style Transfer Supervised method Unsupervised method Style S’ Target T Result T’ Background
- 13. Outline Background / 005 Statistic Based Text Stylization / 014 Context-Aware Unsupervised Text Stylization / 053 Text Effects Transfer via Stylization & Destylization / 077 Text-Centric Image Style Transfer
- 14. STRUCT Group014 Supervised Text Stylization Awesome Typography: Statistics-Based Text Effects Transfer Shuai Yang, Jiaying Liu, Zhouhui Lian, Zongming Guo, CVPR 2017 Statistics-Based Text Effects Transfer
- 15. STRUCT Group015 Supervised Text Stylization Awesome Typography: Statistics-Based Text Effects Transfer Shuai Yang, Jiaying Liu, Zhouhui Lian, Zongming Guo, CVPR 2017 Statistics-Based Text Effects Transfer
- 16. STRUCT Group016 Statistics-Based Text Effects Transfer Problem: Text Effects Transfer Input: S, S', T Output: T ' Source text S Source effects S' Our result T'Target text T Input Output
- 17. STRUCT Group017 Statistics-Based Text Effects Transfer Problem Analysis Motivation Manually design: Require skills & Take too much time Automatic text effects transfer By Algorithms (seconds) By Manual work (hours) Determine color styles, Warp textures to match text shapes Adjust the transparency for visual plausibleness, etc.
- 18. STRUCT Group018 Problem Analysis Challenges Extreme diversity of the text effects and characters Complicated composition of style elements Texture-by-number methods ✘ Simpleness of guidance images Deep-based methods ✘ Complicated composition Plain guidance images texture-by-number Image Analogies (2001) deep-based Neural Doodles (2016) Our result Statistics-Based Text Effects Transfer
- 19. STRUCT Group019 Problem Analysis Key Observation: Distance-based characteristics Textures with similar distances share similar patterns Distance: to text skeleton Pattern: Color and Scale Different patch colors represent different distances to the text skeleton (white) Statistics-Based Text Effects Transfer
- 20. STRUCT Group Texture Effects Statistics Calculation Distance Optimal patch scale Text Effects Generation Multiscale Texture distribution Naturalness Relationship 020 Statistics-Based Texture Effects Transfer Statistics-Based Text Effects Transfer flame denim fabric lava neon
- 21. STRUCT Group rank(q) r(q) 021 Text Effects Statistics Estimation Distance Estimation (a) Text image (b) Radius and distance (c) Text skeleton (d) Radius and its rank (e) Normalized distance Statistics-Based Text Effects Transfer Normalized distance calculation
- 22. STRUCT Group022 Text Effects Statistics Estimation Optimal Patch Scale Detection Fix patch size and resize image to realize multiple scales Determine from large scale to small scale Determination criterion: Statistics-Based Text Effects Transfer Text shape Texture
- 23. STRUCT Group023 Text Effects Statistics Estimation Relation Between Optimal Patch Scale and Distance Histogram Joint probability Posterior probability (for l being the appropriate scale to depict the patches with distances corresponding to bin(q)) Statistics-Based Text Effects Transfer q q qp ~
- 24. STRUCT Group024 Text Effect Transfer Objective Function Eapp Edist Epsy Appearance Term Texture Style Transfer Low-level objective evaluation Distribution Term Spatial Style Transfer Mid-level objective evaluation Psycho-Visual Term Naturalness Preservation subjective evaluation Statistics-Based Text Effects Transfer p: pixel in source images q: pixel in target images
- 25. STRUCT Group025 Appearance Term Text Effect Transfer Statistics-Based Text Effects Transfer 𝑃/𝑃′ : patch in 𝑆/S′ centered at p 𝑄/𝑄′: patch in 𝑇/𝑇′centered at q Text effects term Ensure texture transfer Text shape term Constrain text shape
- 26. STRUCT Group026 Appearance Term single-scale 5*5 single-scale 15*15 multi-scale 5*5 Weighted by posterior probability Patches in different regions use different scales Small patch: Lose structures Large patch: Lose textures Multi-scale: Preserve texture structures Preserve texture details Text Effect Transfer Statistics-Based Text Effects Transfer
- 27. STRUCT Group027 Distribution Term Encourage the text effects of T' to share similar distribution of S' Used for initialization no Edist with Edist with Edist + initialization Text Effect Transfer Statistics-Based Text Effects Transfer
- 28. STRUCT Group028 Psycho-Visual Term |Φ(q)|: Number of patches that find Q(q) as their matched patch Penalize texture over-repetitiveness Encourage new texture creation Weights for Epsy Text Effect Transfer Statistics-Based Text Effects Transfer
- 29. STRUCT Group029 Text Effect Transfer Transfer for words Stroke Term Estr Control the texture similarity of similar strokes Positive weight: stylize in a more consistent way Negative weight: transfer more diverse textures Positive weight Negative weight Statistics-Based Text Effects Transfer
- 30. STRUCT Group030 Statistics-Based Text Effects Transfer Comparison with Other Methods Target Text
- 31. STRUCT Group031 Statistics-Based Text Effects Transfer Comparison with Other Methods A. Hertzmann, C. E. Jacobs, N. Oliver, B. Curless, and D. H. Salesin.. “Image analogies”. SIGGRAPH. 2001. Image Analogy
- 32. STRUCT Group032 Statistics-Based Text Effects Transfer Comparison with Other Methods O. Frigo, N. Sabater, J. Delon, and P. Hellier. Split and match: example-based adaptive patch sampling for unsupervised style transfer”. CVPR. 2016. Split and Match
- 33. STRUCT Group033 Statistics-Based Text Effects Transfer Comparison with Other Methods A. J. Champandard. “Semantic style transfer and turning two-bit doodles into fine artworks”. Arxiv. 2016. Neural Doodles
- 34. STRUCT Group034 Statistics-Based Text Effects Transfer Comparison with Other Methods Our method
- 35. STRUCT Group035 Comparison with Other Methods (a) (S,S') (b) T (c) Image Analogies1 (d) Split & Match2 (e) Neural Doodle3 (f ) Baseline (g) Proposed 3A. J. Champandard. “Semantic style transfer and turning two-bit doodles into fine artworks”. Arxiv. 2016. 2O. Frigo, N. Sabater, J. Delon, and P. Hellier. Split and match: example-based adaptive patch sampling for unsupervised style transfer”. CVPR. 2016. 1A. Hertzmann, C. E. Jacobs, N. Oliver, B. Curless, and D. H. Salesin. “Image analogies”. SIGGRAPH. 2001. Statistics-Based Text Effects Transfer
- 36. STRUCT Group036 Statistics-Based Text Effects Transfer User Study Score distribution 91 participants, give 1-5 scores (5 means best) Averaged scores: 3.04, 1.54, 1.68, 3.95, 4.79
- 37. STRUCT Group037 Statistics-Based Text Effects Transfer User Study 91 participants, give 1-5 scores (5 means best) Highest score over all 8 test images Score of each test image group
- 38. STRUCT Group038 Statistics-Based Text Effects Transfer User Study 91 participants, give 1-5 scores (5 means best) Clear advantages: highly regular texture effects Score of each test image group
- 39. STRUCT Group039 Statistics-Based Text Effects Transfer User Study 91 participants, give 1-5 scores (5 means best) Less obvious advantages: simple effects, fail to cover all colors Score of each test image group
- 40. STRUCT Group040 Statistics-Based Text Effects Transfer Diversified Transfer Apply different text effects to representative characters (Chinese, alphabetic, handwriting)
- 46. STRUCT Group046 Statistics-Based Text Effects Transfer Typography Library Generation
- 47. STRUCT Group047 Statistics-Based Text Effects Transfer Typography Library Generation
- 48. STRUCT Group048 Statistics-Based Text Effects Transfer Typography Library Generation
- 49. STRUCT Group049 Statistics-Based Text Effects Transfer Typography Library Generation
- 50. STRUCT Group050 Statistics-Based Text Effects Transfer Stroke-Based Graphics Rendering Text Icons
- 51. STRUCT Group051 Statistics-Based Text Effects Transfer Stroke-Based Graphics Rendering Icons Icons Icons Text
- 52. Outline Background / 005 Statistic Based Text Stylization / 014 Context-Aware Unsupervised Text Stylization / 053 Text Effects Transfer via Stylization & Destylization / 077 Text-Centric Image Style Transfer
- 53. STRUCT Group53 Unsupervised Text Stylization Context-Aware Unsupervised Text Stylization Shuai Yang, Jiaying Liu, Wenhan Yang, Zongming Guo, ACM MM 2018 / TIP 2019 Context-Aware Unsupervised Text Stylization
- 54. STRUCT Group54 Context-Aware Unsupervised Text Stylization Unsupervised Text Stylization Unsupervised Text Stylization Texture image S' without guidance S S' arbitrary texture images Broaden application scope
- 55. STRUCT Group55 Context-Aware Unsupervised Text Stylization Unsupervised Text Stylization Context-Aware Layout Design Photo + text: poster, magazine cover, billboard Optimal text layout selection Seamlessly embedding Application: posterBackground photo (T, S') photo + text
- 56. STRUCT Group56 Context-Aware Unsupervised Text Stylization Context-Aware Unsupervised Text Stylization Unsupervised Text Stylization Structure transfer Texture transfer Context-Aware Layout Design Position estimation Text embedding narrow the visual discrepancy
- 57. STRUCT Group57 Context-Aware Unsupervised Text Stylization Unsupervised Text Stylization Guidance Map Extraction: Clustering Structure Transfer Texture Transfer Guidance Map Extraction 𝑇 𝑆′ 𝑆 • Unify modality • Establish mapping
- 58. STRUCT Group58 Context-Aware Unsupervised Text Stylization Unsupervised Text Stylization Guidance Map Extraction Structure Transfer Texture Transfer Output 𝑇 𝑆′ 𝑆 Guidance Map Extraction Texture Transfer
- 59. STRUCT Group59 Context-Aware Unsupervised Text Stylization Unsupervised Text Stylization Guidance Map Extraction Structure Transfer: Minimize Structural Inconsistency Texture Transfer Structure Transfer Texture Transfer Output 𝑇𝑇 𝑆′ 𝑆 Guidance Map Extraction
- 60. STRUCT Group60 Context-Aware Unsupervised Text Stylization Unsupervised Text Stylization Guidance Map Extraction Structure Transfer: Legibility-Preserving Structure Transfer Texture Transfer 1A. Rosenberger, D. Cohen-Or, and D. Lischinski. Layered shape synthesis: automatic generation of control maps for non-stationary textures. TOG. 2009. LSS result output bicubic Alg. BW T
- 61. STRUCT Group61 Context-Aware Unsupervised Text Stylization Unsupervised Text Stylization Guidance Map Extraction Structure Transfer: Legibility-Preserving Structure Transfer Texture Transfer 1A. Rosenberger, D. Cohen-Or, and D. Lischinski. Layered shape synthesis: automatic generation of control maps for non-stationary textures. TOG. 2009. T LSS result Our result bicubic Alg. BW output
- 62. STRUCT Group62 Context-Aware Unsupervised Text Stylization Unsupervised Text Stylization Texture Transfer Objective Function Low-level Objective Local texture Mid-level Objective Text boundary Naturalness Subjective Global distribution 1 2 3min ( , ) ( , ) ( , ) ( , )app dist psy sal q p E p q E p q E p q E p q Eapp Edist Epsy Esal Legibility Subjective Non-boundary part Appearance Term Distribution Term Psycho-Visual Term Saliency Term
- 63. STRUCT Group63 Context-Aware Unsupervised Text Stylization Texture Transfer 外观项 Saliency Term S’ Saliency map of S’ λ3=0.00 λ3=0.01 λ3=0.05 Internal area: salient textures External area: indistinctive flat textures Increase text legibility
- 64. STRUCT Group64 Context-Aware Unsupervised Text Stylization Context-Aware Layout Design Position Estimation Cost function Local variance Seek flat region Low saliency Prevent occlusion Texture consistency Prevent seams Uvar Usal Ucoh Uaes Centrality of text Prevent corners ˆ arg min ( ) ( ) ( ) ( )var sal coh aes R p R R U p U p U p U p Variance Cost Saliency Cost Coherence Cost Aesthetics Cost
- 65. STRUCT Group65 Context-Aware Unsupervised Text Stylization Context-Aware Layout Design Position Estimation Cost function Variance Cost Local variance Seek flat region Saliency Cost Low saliency Prevent occlusion Coherence Cost Texture consistency Prevent seams Aesthetics Cost Centrality of text Prevent corners Input T , S′ and I Total placement cost Position estimation result
- 66. STRUCT Group66 Context-Aware Unsupervised Text Stylization Context-Aware Layout Design Text Embedding Image inpainting: fill Inpainting region with samples from S′ Structure guidance of T + Contextual boundary constraint Contextual region Inpainting region Sampling region S′
- 67. STRUCT Group67 Context-Aware Unsupervised Text Stylization Comparison with Other Methods 2A. J. Champandard. Semantic style transfer and turning two-bit doodles into fine artworks. Arxiv. 2016. 3S. Yang, J. Liu, Z. Lian, and Z. Guo. Awesome typography: statistics-based text effects transfer. CVPR. 2017. 1A. Hertzmann, C. E. Jacobs, N. Oliver, B. Curless, and D. H. Salesin. Image analogies. SIGGRAPH. 2001. T and S′ Image Analogy1 Neural Doodle2 Text Effects Transfer3 Image Quilting4 Neural Style5 CNNMRF6 Our method 4A. Efros and W. T. Freeman. Image quilting for texture synthesis and transfer. TOG. 2001. 5L. A. Gatys, A. S. Ecker, and M. Bethge. Image style transfer using convolutional neural networks. CVPR. 2016 6C. Li and M. Wand. Combining Markov random fields and convolutional neural networks for image synthesis. CVPR. 2016
- 68. STRUCT Group68 Context-Aware Unsupervised Text Stylization Comparison with Unsupervised Methods Structural Consistency 4A. Efros and W. T. Freeman. Image quilting for texture synthesis and transfer. TOG. 2001. 5L. A. Gatys, A. S. Ecker, and M. Bethge. Image style transfer using convolutional neural networks. CVPR. 2016 Image Quilting4 Neural Style5 Our method S′ T Structure transfer result
- 69. STRUCT Group69 Context-Aware Unsupervised Text Stylization Comparison with Supervised Methods Text Legibility 3S. Yang, J. Liu, Z. Lian, and Z. Guo. Awesome typography: statistics-based text effects transfer. CVPR. 2017. 1A. Hertzmann, C. E. Jacobs, N. Oliver, B. Curless, and D. H. Salesin. Image analogies. SIGGRAPH. 2001. Our method Image Analogy1 Text Effects Transfer3 T and S′
- 70. STRUCT Group70 Context-Aware Unsupervised Text Stylization Different Fonts and Languages Visual effects for text stylization on different languages. Visual effects for text stylization on different fonts.
- 71. STRUCT Group71 Unsupervised Text Effects Synthesis Text effects synthesis results Context-Aware Unsupervised Text Stylization
- 72. STRUCT Group72 Unsupervised Text Effects Synthesis Application: texture rendering for symbols Context-Aware Unsupervised Text Stylization
- 73. STRUCT Group73 Graphical Arts: Text + Images Synthesize stylish text onto photos Follow image completion process Background images provide boundary information Texture synthesis under boundary constrains Application: Poster design Background image (T, S') Text + Image Context-Aware Unsupervised Text Stylization
- 74. STRUCT Group74 Graphical Arts: Text + Images Graphical arts design results (1/2) S' is sampled from background images Context-Aware Unsupervised Text Stylization
- 75. STRUCT Group75 Graphical Arts: Text + Images Graphical arts design results (2/2) S' is not from background images: color transfer is used Context-Aware Unsupervised Text Stylization
- 76. Outline Background / 005 Statistic Based Text Stylization / 014 Context-Aware Unsupervised Text Stylization / 053 Text Effects Transfer via Stylization & Destylization / 077 Text-Centric Image Style Transfer
- 77. STRUCT Group77 Deep Learning Based Text Stylization Text Effects Transfer via Stylization and Destylization Shuai Yang, Jiaying Liu, Wenjin Wang, Zongming Guo, AAAI 2019 Text Effects Transfer via Stylization & Destylization
- 78. STRUCT Group78 Problem: Text Stylization and Destylization Task1: Stylization for transferring the visual effects from highly stylized text onto other glyphs Task2: Destylization for removing style features from text Text Effects Transfer via Stylization & Destylization
- 79. STRUCT Group79 Problem Analysis Methods Text effects are highly structured along the glyph Global-Based Local-Based GAN Need no dataset Preserve texture details Need no dataset Reconstruct vivid styles Text effects cannot be simply characterized as the mean, variance, covariance and other global statistics Fail to characterize global distribution Need dataset for training Deals only two domains or Hard to extend to new style pros cons e.g. Text Effects Transfer via Stylization & Destylization
- 80. STRUCT Group80 Problem Analysis • 64 different style • 775 Chinesecharacters • 52 English letters • 10 Arabic numerals • Featuredisentanglement • Featurerecombination • One-shot learning METHODDATA APPLICATION • Stylization • Destylization • Style creation flame 字 combine disentangledisentangle Text Effects Transfer via Stylization & Destylization
- 81. STRUCT Group81 Data Overview Text Effects Transfer via Stylization & Destylization Dataset 64 different text effects 775 Chinese characters, 52 English letters, 10 Arabic numerals 64 * 837 ≈ 54K pairs Train : Test = 708 : 129
- 82. STRUCT Group Learn a two-way mapping between two visual domains X (text) and Y (text effects) 82 Network Architecture Text Effects Transfer via Stylization & Destylization • Encoder: Text content encoder Style content encoder Style encoder • Decoder: Content decoder Style decoder • Discriminator: Content discriminator Style discriminator
- 83. STRUCT Group83 Network Architecture Text Effects Transfer via Stylization & Destylization Autoencoder Reconstruction loss Preserve the core information of the glyph
- 84. STRUCT Group84 Network Architecture Text Effects Transfer via Stylization & Destylization Destylization Feature loss Approach ground truth content feature extracted by Ex Pixel loss Approach ground truth output Adversarial loss WGAN-GP Improve the quality
- 85. STRUCT Group85 Network Architecture Text Effects Transfer via Stylization & Destylization Stylization Pixel loss Approach ground truth output Adversarial loss WGAN-GP Improve the quality
- 86. STRUCT Group86 One-shot fine-tuning (supervised) Text Effects Transfer via Stylization & Destylization Extend to new style with only one example pair Random crop to generate multiple training pairs More precise texture details (a) User-specified new text effects. (b) Result on an unseen style (c) Result after one-shot fine-tuning
- 87. STRUCT Group87 One-shot fine-tuning (unsupervised) Text Effects Transfer via Stylization & Destylization Extend to new style with only one style image Use destylization network to generate auxiliary raw text Style autoencoder reconstruction loss Input style Input glyph Auxiliary glyph Destylization network
- 88. STRUCT Group88 Comparison GAN-based Local-based Global-based Text Effects Transfer via Stylization & Destylization
- 89. STRUCT Group89 Comparison One-shot supervised style transfer Fine-tune Train from scratch Pretraining on our dataset successfully teaches our network the domain knowledge of text effects synthesis Text Effects Transfer via Stylization & Destylization
- 90. STRUCT Group90 Comparison One-shot unsupervised style transfer Text Effects Transfer via Stylization & Destylization
- 91. STRUCT Group91 Other results Style interpolation Text Effects Transfer via Stylization & Destylization
- 92. STRUCT Group92 Conclusion Conclusion Supervised Patch-based Deep-based Input Method Statistics-based Text Effect Transfer • Distribution-based prior • Uniform Objective function Unsupervised Text Stylization • Structure + texture transfer • Combine text and images TET-GAN • Feature disentanglement • Data collection • One-shot supervised style transfer • One-shot unsupervised style transfer Unsupervised style glyph style glyph