Resampling
Download as PPTX, PDF•2 likes•1,407 views
This document discusses various methods of image resampling, including downsampling, upsampling, and fractional resampling. It describes traditional blind resampling methods like nearest neighbor, bilinear, bicubic, and Lanczos interpolation. It also covers content-aware resampling techniques like seam carving and new edge-directed interpolation (NEDI) that aim to minimize information loss. For each method, it explains the approach, artifacts, filter kernels, and provides visual examples comparing the output images. The document concludes by discussing tradeoffs between methods and opportunities for future improvement in content-aware resizing.
![28 October 2015 Department of Electronics and Communications, NMAMIT, Nitte.
21
Content Aware Resampling –
New Edge Directed interpolation (NEDI)
• Principle: Inter pixel relations are retained while Downscaling. Hence
Covariance in original Image and Downscaled image are nearly same.
• New Pixel Value = Weighted Sum of nearest 4 pixels. Weights are computed
dynamically according to local image characteristics.
𝑌2𝑖+1,2𝑗+1 =
𝑘=0
1
𝑙=0
1
𝛼2𝑘+𝑙 𝑌2 𝑖+𝑘 ,2 𝑗+𝑙
𝛼 = 𝑅−1
𝑟
𝑅 =
1
22
𝐶 𝑇
𝐶, 𝑎𝑛𝑑 𝑟 =
1
22
𝐶 𝑇
𝑦
Where 𝑦 = [𝑦1 … 𝑦 𝑘 … 𝑦22] 𝑇
is the data vector
containing the 2x2 pixels inside the local window
and C is a 4x22 data matrix whose kth column
vector is the four nearest neighbors of along the
diagonal direction.](https://image.slidesharecdn.com/resampling-151028180232-lva1-app6892/85/Resampling-21-320.jpg)
Resampling
- 1. Tejus Adiga M Department of Electronics and Communication, NMAMIT, Nitte. Presented By:
- 2. Resampling • Ideal resampling: Discrete -> Continuous -> Discrete. • Practical resampling: Done entirely in discrete domain. • Types of Resampling: • Downsampling: Decrease size by M. • Upsampling: Increase size by N. • Fractional Resampling: Increase size my M and decrease by N (M/N). • Traditional Methods: • Blind Resampling: 2D Convolution. Eg Kernels Nearest Neighbor, Bilinear, Bicubic, Bspline. • Content Aware Resampling: Seam Carving, Edge Directed Interpolation (EDI), Super Resolution. • Seperability: 2D filtering = Performing 1D filtering two times in each dimension one after another. 28 October 2015 Department of Electronics and Communications, NMAMIT, Nitte. 2
- 3. 28 October 2015 Department of Electronics and Communications, NMAMIT, Nitte. 3 Downsampling • General Approach: Anti Alias Filter (LPF) Downsampler ↓ 𝑁 Image 𝑊𝑥𝐻 Downsampled Image 𝑊 2 𝑥 𝐻 2 • Practical Approach 𝑦 𝑚 = 𝑘=−𝑁/2 𝑁/2 𝐶 𝑘 𝑥2𝑚 −𝑘−1 0 ≤ 𝑚 ≤ 𝑊, 𝐻. 𝑁 𝑖𝑠 𝑓𝑖𝑙𝑡𝑒𝑟 𝑙𝑒𝑛𝑔𝑡ℎ Downsampling Kernel Downsampled Image 𝑊 2 𝑥 𝐻 2 Image 𝑊𝑥𝐻 • Convolution m-2 m-1 m m+1 m+2 m+3 Downscaled Image Original Image Fig 1: Downsampling process x Ck1Ck1 Ck2Ck2 hd(x) 0 1 Pixel Distance Fig 2: Downsampling Kernel
- 4. 28 October 2015 Department of Electronics and Communications, NMAMIT, Nitte. 4 Upsampling • General Approach: Upsampler ↑ 𝑀 Low Pass FilterImage 𝑊𝑥𝐻 Upsampled Image 2𝑊 𝑥 2𝐻 • Practical Approach 𝑦2𝑚+1 = 𝑘=−𝑁/2 𝑁/2 𝐶 𝑘 𝑥 𝑚 −𝑘−1 0 ≤ 𝑚 ≤ 𝑊, 𝐻. 𝑁 𝑖𝑠 𝑓𝑖𝑙𝑡𝑒𝑟 𝑙𝑒𝑛𝑔𝑡ℎ Upsampling Kernel Upsampled Image 2𝑊 𝑥 2𝐻 Image 𝑊𝑥𝐻 • Convolution Upscaled Image m-2 m-1 m m+1 m+2 m+3 Original Image m-3 m+4 Fig 3: Upsampling process hu(k) x 0 Fig 4: Upsampling Kernel
- 5. 28 October 2015 Department of Electronics and Communications, NMAMIT, Nitte. 5 Blind Resampling – Nearest Neighbor Fig 5: Spatial kernel and Frequency Response • Downsampling: Discard Every alternate pixel. • Upsampling: Replicate the Nearest Pixel. • Artifacts: Aliasing-Increase 4 times for two fold resample. • Kernel: Rectangular spatial kernel. Infinite frequency contents.
- 6. 28 October 2015 Department of Electronics and Communications, NMAMIT, Nitte. 6 Blind Resampling – Nearest Neighbor Downsampled by 4 Downsampled by 2 Captured Image Upsampled by 4 Fig 6: Downsampled and Upsampled by factor of 2 and 4 Upsampled by 2
- 7. 28 October 2015 Department of Electronics and Communications, NMAMIT, Nitte. 7 Blind Resampling – Bilinear Fig 7: Spatial kernel and Frequency Response • Downsampling and Upsampling: Average of two pixels. (4 pixels in 2D) • Artifacts: Aliasing, Blurring. • Filter Coefficients: ℎ 𝑏 𝑥 = 1 2 , 1 2 2 = 0.5, 0.5 • Kernel: Triangular or Tent Spatial kernel. • Frequency response: Stop band attenuation better than Nearest Neighbor. • Aliasing is reduced when compared to nearest Neighbor.
- 8. 28 October 2015 Department of Electronics and Communications, NMAMIT, Nitte. 8 Blind Resampling – Bilinear Captured Image Upsampled by 4 Fig 8: Downsampled and Upsampled by factor of 2 and 4 Upsampled by 2 Downsampled by 4 Downsampled by 2
- 9. 28 October 2015 Department of Electronics and Communications, NMAMIT, Nitte. 9 Blind Resampling – Bicubic Fig 9: Spatial kernel and Frequency Response • Downsampling and Upsampling: Weighted average of 4 pixels. • Artifacts: Blurring. • Filter: ℎ 𝑏 𝑥 = 3 2 𝑥 3 − 5 2 𝑥 2 + 1, 0 ≤ 𝑥 ≤ 1 − 1 2 𝑥 3 + 5 2 𝑥 2 − 4 𝑥 + 2, 1 < 𝑥 ≤ 2 0, 𝑥 > 2 • Frequency response: Stop band attenuation better than Bilinear. • The 1st negative side lobe introduce controlled Ringing effect which makes image appear sharper than they actually are.
- 10. 28 October 2015 Department of Electronics and Communications, NMAMIT, Nitte. 10 Blind Resampling – Bicubic Captured Image Upsampled by 4 Fig 10: Downsampled and Upsampled by factor of 2 and 4 Upsampled by 2 Downsampled by 4 Downsampled by 2
- 11. 28 October 2015 Department of Electronics and Communications, NMAMIT, Nitte. 11 Blind Resampling – Windowed Sinc Fig 11: Windowed Sinc Kernel • Artifacts: Ringing, Blurring. • Filter: Truncated Sinc function. ℎ 𝑏 𝑥 = sin 𝜋𝑥 2 𝜋𝑥 2 • Side lobes significantly contributes to Ringing.
- 12. 28 October 2015 Department of Electronics and Communications, NMAMIT, Nitte. 12 Blind Resampling – Lanczos Fig 12: Lanczos Kernel for a=2 and a=3 • Weighted Average of 4 pixels. • Artifacts: Blurring. • Filter: ℎ 𝑏 𝑥 = 𝑠𝑖𝑛𝑐 𝑥 𝑠𝑖𝑛𝑐 𝑥 𝑎 − 𝑎 ≤ 𝑥 ≤ 𝑎 𝑎𝑛𝑑 𝑎 = 2, 3 • ‘a’ indicates number of lobes in one half of the filter. • Effect of side lobes is decreased by multiplying another scaled sinc function. But stronger enough to make image look sharper. • Upsampled image sharper than Bicubic, Bilinear.
- 13. 28 October 2015 Department of Electronics and Communications, NMAMIT, Nitte. 13 Blind Resampling – Lanczos Captured Image Upsampled by 4 Fig 13: Downsampled and Upsampled by factor of 2 and 4 Upsampled by 2 Downsampled by 4 Downsampled by 2
- 14. 28 October 2015 Department of Electronics and Communications, NMAMIT, Nitte. 14 Artifacts in Image Resampling • Aliasing: • Jagged Edges. • Introduced in Downsampling and Enhanced in Upsampling. • Priority: Lanczos, Bicubic, Bilinear, nearest Neighbor. • Ringing: • Side lobes of lengthy filter contribute to false edges. • Optimal filter length 4. • Windowed Sinc. • Blurring: • LPF gains get multiplied while Downsampling and Upsampling. • Information lost during Downsampling is irreversible. So in upsampling pixels are filled with he help of existing information in Downsampled image. • Priority: Nearest Neighbor, Bilinear, Lanczos, Bicubic.
- 15. 28 October 2015 Department of Electronics and Communications, NMAMIT, Nitte. 15 Artifacts Ringing Example Aliasing Example Blurring Example Fig 14: Examples of Ringing, Aliasing and Blurring
- 16. 28 October 2015 Department of Electronics and Communications, NMAMIT, Nitte. 16 Content Aware Resampling – Seam Carving • Seam: 8-Connected set of pixels that runs from top to bottom or Left to Right. • Principle: Low energy seam is not appealing to eyes. • Applications: • Image Retargeting: Resizing image, Changing Aspect Ratio. • Object removal or insertion. • Algorithm: • Find the Gradient Map of the input image I. 𝐺 𝑥, 𝑦 = 𝜕𝐼 𝜕𝑥 + 𝜕𝐼 𝜕𝑦 • In Gradient Map search a unique path (seam) from top to bottom or left to right such that Energy of the seam is minimum than all other possible seam. 𝑠 = min 𝑠 𝑊/𝐻 𝑔(𝑥, 𝑦) • Remove the seam or Duplicate the seam from the image I and G which reduces/increases the width/height by 1 pixel. • Iterate the above steps until desired size is achieved.
- 17. 28 October 2015 Department of Electronics and Communications, NMAMIT, Nitte. 17 Content Aware Resampling – Seam Carving Fig 15: Seam Calculation using Gradient.
- 18. 28 October 2015 Department of Electronics and Communications, NMAMIT, Nitte. 18 Content Aware Resampling – Seam Carving Fig 16: Comparison of Seam Carving with Scaling
- 19. 28 October 2015 Department of Electronics and Communications, NMAMIT, Nitte. 19 Seam Carving – Failure Cases
- 20. 28 October 2015 Department of Electronics and Communications, NMAMIT, Nitte. 20 Seam Carving – Failure Cases
- 21. 28 October 2015 Department of Electronics and Communications, NMAMIT, Nitte. 21 Content Aware Resampling – New Edge Directed interpolation (NEDI) • Principle: Inter pixel relations are retained while Downscaling. Hence Covariance in original Image and Downscaled image are nearly same. • New Pixel Value = Weighted Sum of nearest 4 pixels. Weights are computed dynamically according to local image characteristics. 𝑌2𝑖+1,2𝑗+1 = 𝑘=0 1 𝑙=0 1 𝛼2𝑘+𝑙 𝑌2 𝑖+𝑘 ,2 𝑗+𝑙 𝛼 = 𝑅−1 𝑟 𝑅 = 1 22 𝐶 𝑇 𝐶, 𝑎𝑛𝑑 𝑟 = 1 22 𝐶 𝑇 𝑦 Where 𝑦 = [𝑦1 … 𝑦 𝑘 … 𝑦22] 𝑇 is the data vector containing the 2x2 pixels inside the local window and C is a 4x22 data matrix whose kth column vector is the four nearest neighbors of along the diagonal direction.
- 22. 28 October 2015 Department of Electronics and Communications, NMAMIT, Nitte. 22 Content Aware Resampling – (NEDI) Fig 16: Comparison of NEDI with Bicubic filter Downscaled Image Upscaled 4X using NEDI Upscaled 4X using Bicubic
- 23. 28 October 2015 Department of Electronics and Communications, NMAMIT, Nitte. 23 Conclusion • Trade off: • Quality • Speed of Operation • Requirement • Information lost during Downsampling cannot be recovered while upsampling. • Future Work: • Improvement of Content Aware Resizing methods. • Adding Resolution.
- 24. 28 October 2015 Department of Electronics and Communications, NMAMIT, Nitte. 24 References 1. New Edge-Directed Interpolation, Xin Li and Michael T. Orchard. IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 10, NO. 10, OCTOBER 2012. 2. Local and Nonlocal Regularization to Image Interpolation, Yi Zhan, Sheng Jie Li, and Meng Li, Mathematical Problems in Engineering, Volume 2014 (2014), Article ID 230348. 3. Adaptive multidirectional edge directed interpolation for selected edge regions. TENCON 2011 - 2011 IEEE Region 10 Conference. 4. V.R. Algazi, G.E. Ford and R. Potharlanka, "Directional interpolation of images based on visual properties and rank order filtering", Proceeding of ICASSP' 1991, pp.3005-3008. 5. Seam Carving for Content-Aware Image Resizing. Shai Avidan and Ariel Shamir. Proceedings of ACM SIGGRAPH, 417–424. 6. J. Allebach and P.W. Wong, "Edge-directed interpolation", Proceeding of ICIP 1996, Page No 707-710. 7. Keys, R., “Cubic Convolution Interpolation for Digital Image Processing”, IEEE Trans on ASSP, vol ASSP-29, No. 6, Page No 1153-1160. Dec 1981. 8. New Filters for Image Interpolation and Resizing, Amir Said, IEEE International Conference on Image Processing, VOL. 8, 2007. 9. Image Zooming Methods, Bax Smith. 10. “Interpolation Theory” http://sepwww.stanford.edu/public/docs/sep107/paper_html/node20.html