Filtering in digital signal and image processing
Download as PPTX, PDF0 likes27 views
The document discusses various methods for image filtering, focusing on the application of linear and non-linear filters including box and Gaussian filters. It outlines the mathematical foundations of these filters, their properties, and the implications of different padding techniques when processing images. Additionally, it examines runtime complexity and addresses challenges such as noise reduction and outlier distortion in filtered images.
![Non-linear Filters (2D)
[040, 081, 013, 125, 830, 076, 144, 092, 108]
92
Sort
[013, 040, 076, 081, 092, 108, 125, 144, 830]
[830, 076, 080, 092, 108, 095, 102, 106, 087]
[076, 080, 087, 092, 095, 102, 106, 108, 830]
Sort
95
40 81
125 830
144 92
13
76
108
22
80
95
132 102 106 87](https://image.slidesharecdn.com/05filtering-241123005747-93a7231b/85/Filtering-in-digital-signal-and-image-processing-51-320.jpg)
![Is Median Filtering Linear?
Example from (I believe): Kristen Grauman
[
1 1 1
1 1 2
2 2 2] [
0 0 0
0 1 0
0 0 0]
+
[
1 1 1
1 2 2
2 2 2]
=
Median Filter
1 0 2
+ =](https://image.slidesharecdn.com/05filtering-241123005747-93a7231b/85/Filtering-in-digital-signal-and-image-processing-54-320.jpg)
Filtering in digital signal and image processing
- 1. Filtering
- 2. Let’s Take An Image
- 3. Let’s Fix Things Slide Credit: D. Lowe • We have noise in our image • Let’s replace each pixel with a weighted average of its neighborhood • Weights are filter kernel 1/9 1/9 1/9 1/9 1/9 1/9 1/9 1/9 1/9 Out
- 4. 1D Case 1/3 1/3 1/3 Filter/ David Signal/ Front Row 10 12 9 11 10 11 12 Output 10.33 10.66 10 10.66 11
- 5. Applying a Linear Filter I11 I12 I13 I21 I22 I23 I31 I32 I33 I14 I15 I16 I24 I25 I26 I34 I35 I36 I41 I42 I43 I51 I52 I53 I44 I45 I46 I54 I55 I56 Input F11 F12 F13 F21 F22 F23 F31 F32 F33 Filter O11 O12 O13 O21 O22 O23 O31 O32 O33 O14 O24 O34 Output
- 6. Applying a Linear Filter I11 I12 I13 I21 I22 I23 I31 I32 I33 I14 I15 I16 I24 I25 I26 I34 I35 I36 I41 I42 I43 I51 I52 I53 I44 I45 I46 I54 I55 I56 Input & Filter F11 F12 F13 F21 F22 F23 F31 F32 F33 Output O11 O11 = I11*F11 + I12*F12 + … + I33*F33
- 7. Applying a Linear Filter I11 I12 I13 I21 I22 I23 I31 I32 I33 I14 I15 I16 I24 I25 I26 I34 I35 I36 I41 I42 I43 I51 I52 I53 I44 I45 I46 I54 I55 I56 Input & Filter F11 F12 F13 F21 F22 F23 F31 F32 F33 Output O11 O12 = I12*F11 + I13*F12 + … + I34*F33 O12
- 8. Applying a Linear Filter I11 I12 I13 I21 I22 I23 I31 I32 I33 I14 I15 I16 I24 I25 I26 I34 I35 I36 I41 I42 I43 I51 I52 I53 I44 I45 I46 I54 I55 I56 Input F11 F12 F13 F21 F22 F23 F31 F32 F33 Filter Output How many times can we apply a 3x3 filter to a 5x6 image?
- 9. Applying a Linear Filter I11 I12 I13 I21 I22 I23 I31 I32 I33 I14 I15 I16 I24 I25 I26 I34 I35 I36 I41 I42 I43 I51 I52 I53 I44 I45 I46 I54 I55 I56 Input Output Oij = Iij*F11 + Ii(j+1)*F12 + … + I(i+2)(j+2)*F33 O11 O12 O13 O21 O22 O23 O31 O32 O33 O14 O24 O34 F11 F12 F13 F21 F22 F23 F31 F32 F33 Filter
- 10. Painful Details – Edge Cases f g g g g f g g g g f g g g g full same valid Convolution doesn’t keep the whole image. Suppose f is the image and g the filter. f/g Diagram Credit: D. Lowe Full – any part of g touches f. Same – same size as f; Valid – only when filter doesn’t fall off edge.
- 11. Painful Details – Edge Cases What to about the “?” region? Symm: fold sides over pad/fill: add value, often 0 f g g g g ? ? ? ? Circular/Wrap: wrap around f/g Diagram Credit: D. Lowe
- 12. Painful Details – Does it Matter? Input Image Box Filtered ??? Box Filtered ??? (I’ve applied the filter per-color channel) Which padding did I use and why? Note – this is a zoom of the filtered, not a filter of the zoomed
- 13. Painful Details – Does it Matter? Input Image Box Filtered Symm Pad Box Filtered Zero Pad (I’ve applied the filter per-color channel) Note – this is a zoom of the filtered, not a filter of the zoomed
- 14. Practice with Linear Filters Slide Credit: D. Lowe Original ? 0 0 0 0 1 0 0 0 0
- 15. Practice with Linear Filters Slide Credit: D. Lowe Original 0 0 0 0 1 0 0 0 0 The Same!
- 16. Practice with Linear Filters Slide Credit: D. Lowe Original ? 0 0 0 0 0 1 0 0 0
- 17. Practice with Linear Filters Slide Credit: D. Lowe Original 0 0 0 0 0 1 0 0 0 Shifted LEFT 1 pixel
- 18. Practice with Linear Filters Slide Credit: D. Lowe Original ? 0 1 0 0 0 0 0 0 0
- 19. Practice with Linear Filters Slide Credit: D. Lowe Original 0 1 0 0 0 0 0 0 0 Shifted DOWN 1 pixel
- 20. Practice with Linear Filters ? Slide Credit: D. Lowe Original 1/9 1/9 1/9 1/9 1/9 1/9 1/9 1/9 1/9
- 21. Practice with Linear Filters Slide Credit: D. Lowe Original 1/9 1/9 1/9 1/9 1/9 1/9 1/9 1/9 1/9 Blur (Box Filter)
- 22. Practice with Linear Filters ? Slide Credit: D. Lowe Original 1/9 1/9 1/9 1/9 1/9 1/9 1/9 1/9 1/9 0 0 0 0 2 0 0 0 0 -
- 23. Practice with Linear Filters Slide Credit: D. Lowe Original 1/9 1/9 1/9 1/9 1/9 1/9 1/9 1/9 1/9 0 0 0 0 2 0 0 0 0 - Sharpened (Acccentuates difference from local average)
- 24. Sharpening Slide Credit: D. Lowe
- 25. Properties – Linear Assume: I image f1, f2 filters Linear: apply(I,f1+f2) = apply(I,f1) + apply(I,f2) I is a white box on black, and f1, f2 are rectangles Note: I am showing filters un-normalized and blown up. They’re a smaller box filter (i.e., each entry is 1/(size^2)) = = + =A( , ) + A( , ) = )+A( A( , , )
- 26. Properties – Shift-Invariant Assume: I image, f filter Shift-invariant: shift(apply(I,f)) = apply(shift(I,f)) Intuitively: only depends on filter neighborhood A( , ) = A( , ) =
- 27. Painful Details – Signal Processing Often called “convolution”. Actually cross- correlation. Cross-Correlation (Original Orientation) Convolution (Flipped in x and y)
- 28. Properties of Convolution • Any shift-invariant, linear operation is a convolution ( ) ⁎ • Commutative: f g = g f ⁎ ⁎ • Associative: (f g) h = f (g h) ⁎ ⁎ ⁎ ⁎ • Distributes over +: f (g + h) = f g + f h ⁎ ⁎ ⁎ • Scalars factor out: kf g = f kg = k (f g) ⁎ ⁎ ⁎ • Identity (a single one with all zeros): Property List: K. Grauman = *
- 29. Questions? • Nearly everything onwards is a convolution. • This is important to get right.
- 30. Smoothing With A Box Intuition: if filter touches it, it gets a contribution. Input Box Filter
- 31. Solution – Weighted Combination Intuition: weight contributions according to closeness to center. 𝐹𝑖𝑙𝑡𝑒𝑟𝑖𝑗 ∝1 𝐹𝑖𝑙𝑡𝑒𝑟𝑖𝑗 ∝exp (− 𝑥2 + 𝑦2 2 𝜎 2 ) What’s this?
- 32. Recognize the Filter? 𝐹𝑖𝑙𝑡𝑒𝑟𝑖𝑗 ∝ 1 2𝜋 𝜎 2 exp (− 𝑥 2 +𝑦 2 2 𝜎 2 ) It’s a Gaussian! 0.003 0.013 0.022 0.013 0.003 0.013 0.060 0.098 0.060 0.013 0.022 0.098 0.162 0.098 0.022 0.013 0.060 0.098 0.060 0.013 0.003 0.013 0.022 0.013 0.003
- 33. Smoothing With A Box & Gauss Still have some speckles, but it’s not a big box Input Box Filter Gauss. Filter
- 34. Gaussian Filters σ = 1 filter = 21x21 σ = 2 filter = 21x21 σ = 4 filter = 21x21 σ = 8 filter = 21x21 Note: filter visualizations are independently normalized throughout the slides so you can see them better
- 36. Applying Gaussian Filters Input Image (no filter)
- 37. Applying Gaussian Filters σ = 1
- 38. Applying Gaussian Filters σ = 2
- 39. Applying Gaussian Filters σ = 4
- 40. Applying Gaussian Filters σ = 8
- 41. Picking a Filter Size σ = 8, size = 21 σ = 8, size = 43 Too small filter → bad approximation Want size ≈ 6σ (99.7% of energy) Left far too small; right slightly too small!
- 42. Runtime Complexity for ImageY in range(N): for ImageX in range(N): for FilterY in range(M): for FilterX in range(M): … Time: O(N2 M2 ) I11 I12 I13 I21 I22 I23 I31 I32 I33 I14 I15 I16 I24 I25 I26 I34 I35 I36 I41 I42 I43 I44 I45 I46 I51 I52 I53 I54 I55 I56 F11 F12 F13 F21 F22 F23 F31 F32 F33 I61 I62 I63 I64 I65 I66 Image size = NxN = 6x6 Filter size = MxM = 3x3
- 43. Separability Fy1 Fy2 Fy3 Fx1 Fx2 Fx3 ⁎ = Fx1 * Fy1 Fx1 * Fy2 Fx1 * Fy3 Fx2 * Fy1 Fx2 * Fy2 Fx2 * Fy3 Fx3 * Fy1 Fx3 * Fy2 Fx3 * Fy3 Conv(vector, transposed vector) → outer product
- 44. Separability 𝐹𝑖𝑙𝑡𝑒𝑟𝑖𝑗 ∝ 1 2𝜋 𝜎 2 exp (− 𝑥 2 +𝑦 2 2 𝜎 2 ) 𝐹𝑖𝑙𝑡𝑒𝑟𝑖𝑗 ∝ 1 √2𝜋 𝜎 exp(− 𝑥 2 2 𝜎 2 ) 1 √2 𝜋 𝜎 exp (− 𝑦 2 2 𝜎 2 ) →
- 45. Separability ⁎ = 1D Gaussian 1D Gaussian = 2D Gaussian ⁎ Image 2D Gauss = Image (1D Gauss 1D Gauss ) ⁎ ⁎ ⁎ = (Image 1D Gauss) 1D Gauss ⁎ ⁎
- 46. Runtime Complexity for ImageY in range(N): for ImageX in range(N): for FilterY in range(M): … for ImageY in range(N): for ImageX in range(N): for FilterX in range(M): I11 I12 I13 I21 I22 I23 I31 I32 I33 I14 I15 I16 I24 I25 I26 I34 I35 I36 I41 I42 I43 I44 I45 I46 I51 I52 I53 I54 I55 I56 I61 I62 I63 I64 I65 I66 Image size = NxN = 6x6 Filter size = Mx1 = 3x1 F1 F2 F3 What are my compute savings for a 13x13 filter?
- 47. Why Gaussian? Gaussian filtering removes parts of the signal above a certain frequency. Often noise is high frequency and signal is low frequency.
- 49. Applying Gaussian Filters σ = 1
- 50. Why Does This Fail? 0.1 0.8 0.1 Filter Signal 10 12 9 8 1000 11 10 12 Output 11.5 9.2 107.3 801.9 109.8 10.3 Means can be arbitrarily distorted by outliers What else is an “average” other than a mean?
- 51. Non-linear Filters (2D) [040, 081, 013, 125, 830, 076, 144, 092, 108] 92 Sort [013, 040, 076, 081, 092, 108, 125, 144, 830] [830, 076, 080, 092, 108, 095, 102, 106, 087] [076, 080, 087, 092, 095, 102, 106, 108, 830] Sort 95 40 81 125 830 144 92 13 76 108 22 80 95 132 102 106 87
- 54. Is Median Filtering Linear? Example from (I believe): Kristen Grauman [ 1 1 1 1 1 2 2 2 2] [ 0 0 0 0 1 0 0 0 0] + [ 1 1 1 1 2 2 2 2 2] = Median Filter 1 0 2 + =
- 55. Some Examples of Filtering
- 56. Filtering – Sharpening - Image Smoothed = Details
- 57. Filtering – Sharpening +α Image Details = “Sharpened” α=1
- 58. Filtering – Sharpening = +α Image Details “Sharpened” α=0
- 59. Filtering – Sharpening = +α Image Details “Sharpened” α=2
- 60. Filtering – Sharpening = +α Image Details “Sharpened” α=0
- 61. Filtering – Extreme Sharpening = +α Image Details “Sharpened” α=10
- 62. Filtering -1 0 1 Dx Dy -1 0 1 T What’s this Filter?
- 63. Filtering – Derivatives (Dx2 + Dy2 )1/2
- 64. Filtering – Counting ⁎ = r=10 Pixels Disk ??? How many “on” pixels have 10+ neighbors within 10 pixels?
- 65. Filtering – Counting How many “on” pixels have 10+ neighbors within 10 pixels? x = Pixels Answer Density
- 66. Filtering – Missing Data Oh no! Missing data! (and we know where) Common with many non-normal cameras (e.g., depth cameras)
- 67. Filtering – Missing Data Binary Mask Image ⁎ ⁎ Per-element /
- 68. Filtering – Missing Data Binary Mask Image Per-element /
- 69. Filtering – Missing Data Before
- 70. Filtering – Missing Data After
- 71. Filtering – Missing Data After (without missing data)