SlideShare a Scribd company logo
Distance Functions and  Metric Learning: Part 1  12/1/2011  version. Updated Version: http://www.cs.huji.ac.il/~ofirpele/DFML_ECCV2010_tutorial/ Michael Werman Ofir Pele The Hebrew University of Jerusalem
Rough Outline Bin-to-Bin Distances. Cross-Bin Distances: Quadratic-Form (aka Mahalanobis) / Quadratic-Chi. The Earth Mover’s Distance. Perceptual Color Differences. Hands-On Code Example.
Distance ?
Metric  ² N o n - n e g a t i v i t y : D ( P ; Q ) ¸ 0 ² I d e n t i t y o f i n d i s c e r n i b l e s : D ( P ; Q ) = 0 i ® P = Q ² S y m m e t r y : D ( P ; Q ) = D ( Q ; P ) ² S u b a d d i t i v i t y ( t r i a n g l e i n e q u a l i t y ) : D ( P ; Q ) · D ( P ; K ) + D ( K ; Q )
Pseudo-Metric  (aka Semi-Metric) ² N o n - n e g a t i v i t y : D ( P ; Q ) ¸ 0 ² P r o p e r t y c h a n g e d t o : D ( P ; Q ) = 0 i f P = Q ² S y m m e t r y : D ( P ; Q ) = D ( Q ; P ) ² S u b a d d i t i v i t y ( t r i a n g l e i n e q u a l i t y ) : D ( P ; Q ) · D ( P ; K ) + D ( K ; Q )
Metric  A n o b j e c t i s m o s t s i m i l a r t o i t s e l f ² N o n - n e g a t i v i t y : D ( P ; Q ) ¸ 0 ² I d e n t i t y o f i n d i s c e r n i b l e s : D ( P ; Q ) = 0 i ® P = Q
Metric  ² S y m m e t r y : D ( P ; Q ) = D ( Q ; P ) ² S u b a d d i t i v i t y ( t r i a n g l e i n e q u a l i t y ) : D ( P ; Q ) · D ( P ; K ) + D ( K ; Q ) U s e f u l f o r m a n y a l g o r i t h m s
Minkowski-Form Distances  L p ( P ; Q ) = Ã X i j P i ¡ Q i j p ! 1 p
Minkowski-Form Distances  L 2 ( P ; Q ) = s X i ( P i ¡ Q i ) 2 L 1 ( P ; Q ) = X i j P i ¡ Q i j L 1 ( P ; Q ) = m a x i j P i ¡ Q i j
Kullback-Leibler Divergence Information theoretic origin. Non symmetric. K L ( P ; Q ) = X i P i l o g P i Q i Q i = 0 ?
Jensen-Shannon Divergence J S ( P ; Q ) = 1 2 K L ( P ; M ) + 1 2 K L ( Q ; M ) M = 1 2 ( P + Q ) J S ( P ; Q ) = 1 2 X i P i l o g 2 P i P i + Q i + 1 2 X i Q i l o g 2 Q i P i + Q i
Jensen-Shannon Divergence Information Theoretic origin. Symmetric. is a metric.  J S ( P ; Q ) = 1 2 K L ( P ; M ) + 1 2 K L ( Q ; M ) M = 1 2 ( P + Q ) p J S
Jensen-Shannon Divergence Using Taylor extension and some algebra: J S ( P ; Q ) = 1 X n = 1 1 2 n ( 2 n ¡ 1 ) X i ( P i ¡ Q i ) 2 n ( P i + Q i ) 2 n ¡ 1 = 1 2 X i ( P i ¡ Q i ) 2 ( P i + Q i ) + 1 1 2 X i ( P i ¡ Q i ) 4 ( P i + Q i ) 3 + : : :
Histogram Distance   Statistical origin. Experimentally results are very similar to JS. Reduces the effect of large bins.  is a metric   2  2 ( P ; Q ) = 1 2 X i ( P i ¡ Q i ) 2 ( P i + Q i ) p  2
Histogram Distance   < Â 2 Â 2 ( ; ) Â 2 ( ; )
Histogram Distance   > Â 2 L 1 ( ; ) L 1 ( ; )
Histogram Distance   Experimentally better than  . Â 2 L 2 Â 2 ( P ; Q ) = 1 2 X i ( P i ¡ Q i ) 2 ( P i + Q i )
Bin-to-Bin Distances Bin-to-Bin distances such as  L2   are sensitive to quantization: > L 1 ( ; ) L 1 ; L 2 ; Â 2 L 1 ( ; )
#bins #bins Bin-to-Bin Distances robustness  distinctiveness robustness  distinctiveness  > L 1 ( ; ) L 1 ( ; )
#bins #bins Bin-to-Bin Distances robustness  distinctiveness robustness  distinctiveness  Can we achieve  robustness   and  distinctiveness  ?
The Quadratic-Form Histogram Distance   is the similarity between bin i and j. If  is the inverse of the covariance matrix, QF is called Mahalanobis distance. A i j = s X i j ( P i ¡ Q i ) ( P j ¡ Q j ) A i j ( 1 ) Q F A ( P ; Q ) = q ( P ¡ Q ) T A ( P ¡ Q ) ( 1 ) A
The Quadratic-Form Histogram Distance   Q F A ( P ; Q ) = q ( P ¡ Q ) T A ( P ¡ Q ) = s X i j ( P i ¡ Q i ) ( P j ¡ Q j ) A i j A = I = s X i j ( P i ¡ Q i ) 2 = L 2 ( P ; Q )
The Quadratic-Form Histogram Distance   Does not reduce the effect of large bins. Alleviates the quantization problem. Linear time computation in # non zero  .  A i j Q F A ( P ; Q ) = q ( P ¡ Q ) T A ( P ¡ Q )
The Quadratic-Form Histogram Distance   If A is  positive-semidefinite  then QF is a  pseudo-metric.   Q F A ( P ; Q ) = q ( P ¡ Q ) T A ( P ¡ Q )
Pseudo-Metric  (aka Semi-Metric) ² N o n - n e g a t i v i t y : D ( P ; Q ) ¸ 0 ² P r o p e r t y c h a n g e d t o : D ( P ; Q ) = 0 i f P = Q ² S y m m e t r y : D ( P ; Q ) = D ( Q ; P ) ² S u b a d d i t i v i t y ( t r i a n g l e i n e q u a l i t y ) : D ( P ; Q ) · D ( P ; K ) + D ( K ; Q )
The Quadratic-Form Histogram Distance   If A is  positive-definite  then QF is a metric.  Q F A ( P ; Q ) = q ( P ¡ Q ) T A ( P ¡ Q )
The Quadratic-Form Histogram Distance   Q F A ( P ; Q ) = q ( P ¡ Q ) T A ( P ¡ Q ) = q ( P ¡ Q ) T W T W ( P ¡ Q ) = L 2 ( W P ; W Q )
The Quadratic-Form Histogram Distance   We assume there is a  linear transformation   that makes bins  independent  There are cases where this is  not true e.g.   COLOR Q F A ( P ; Q ) = q ( P ¡ Q ) T A ( P ¡ Q ) = L 2 ( W P ; W Q )
The Quadratic-Form Histogram Distance   Converting distance to similarity (Hafner  et. al  95): Q F A ( P ; Q ) = q ( P ¡ Q ) T A ( P ¡ Q ) A i j = 1 ¡ D i j m a x i j ( D i j )
The Quadratic-Form Histogram Distance   Converting distance to similarity (Hafner  et. al  95): Q F A ( P ; Q ) = q ( P ¡ Q ) T A ( P ¡ Q ) A i j = e ¡ ® D ( i ; j ) m a x i j ( D ( i ; j ) ) I f ® i s l a r g e e n o u g h , A w i l l b e p o s i t i v e - d e ¯ n i t i v e
The Quadratic-Form Histogram Distance   Learning the similarity matrix: part   2 of this tutorial. Q F A ( P ; Q ) = q ( P ¡ Q ) T A ( P ¡ Q )
The Quadratic-Chi Histogram Distance   is the similarity between bin i and j. Generalizes  and  .  Reduces the effect of large bins. Alleviates the quantization problem. Linear time computation in # non zero  .  Â 2 Q C A m ( P ; Q ) = v u u t X i j ( P i ¡ Q i ) ( P j ¡ Q j ) A i j ( P c ( P c + Q c ) A c i ) m ( P c ( P c + Q c ) A c j ) m A i j A i j Q F
The Quadratic-Chi Histogram Distance   is non-negative if A is positive-semidefinite. Symmetric. Triangle inequality unknown. If we define  and  , QC is continuous.  Q C A m ( P ; Q ) = v u u t X i j ( P i ¡ Q i ) ( P j ¡ Q j ) A i j ( P c ( P c + Q c ) A c i ) m ( P c ( P c + Q c ) A c j ) m 0 · m < 1 0 0 = 0
Similarity-Matrix-Quantization-Invariant Property ( , ) D ) ( , = D
Sparseness-Invariant = D ) ( , Empty bins D ( , )
The Quadratic-Chi Histogram Distance  Code f u n c t i o n d i s t = Q u a d r a t i c C h i ( P , Q , A , m ) Z = ( P + Q ) * A ; % 1 c a n b e a n y n u m b e r a s Z _ i = = 0 i f f D _ i = 0 Z ( Z = = 0 ) = 1 ; Z = Z . ^ m ; D = ( P - Q ) . / Z ; % m a x i s r e d u n d a n t i f A i s % p o s i t i v e - s e m i d e f i n i t e d i s t = s q r t ( m a x ( D * A * D ' , 0 ) ) ;
The Quadratic-Chi Histogram Distance  Code f u n c t i o n d i s t = Q u a d r a t i c C h i ( P , Q , A , m ) Z = ( P + Q ) * A ; % 1 c a n b e a n y n u m b e r a s Z _ i = = 0 i f f D _ i = 0 Z ( Z = = 0 ) = 1 ; Z = Z . ^ m ; D = ( P - Q ) . / Z ; % m a x i s r e d u n d a n t i f A i s % p o s i t i v e - s e m i d e f i n i t e d i s t = s q r t ( m a x ( D * A * D ' , 0 ) ) ; T i m e C o m p l e x i t y : O ( # A i j 6 = 0 )
The Quadratic-Chi Histogram Distance  Code f u n c t i o n d i s t = Q u a d r a t i c C h i ( P , Q , A , m ) Z = ( P + Q ) * A ; % 1 c a n b e a n y n u m b e r a s Z _ i = = 0 i f f D _ i = 0 Z ( Z = = 0 ) = 1 ; Z = Z . ^ m ; D = ( P - Q ) . / Z ; % m a x i s r e d u n d a n t i f A i s % p o s i t i v e - s e m i d e f i n i t e d i s t = s q r t ( m a x ( D * A * D ' , 0 ) ) ; W h a t a b o u t s p a r s e ( e . g . B o W ) h i s t o g r a m s ?
The Quadratic-Chi Histogram Distance Code 0 0 0 0 0 0 0 0 0 4 5 0 -3 P ¡ Q = W h a t a b o u t s p a r s e ( e . g . B o W ) h i s t o g r a m s ? T i m e C o m p l e x i t y : O ( S K )
The Quadratic-Chi Histogram Distance Code 0 0 0 0 0 0 0 0 0 4 5 0 -3 P ¡ Q = # ( P 6 = 0 ) + # ( Q 6 = 0 ) W h a t a b o u t s p a r s e ( e . g . B o W ) h i s t o g r a m s ? T i m e C o m p l e x i t y : O ( S K )
The Quadratic-Chi Histogram Distance Code A v e r a g e o f n o n - z e r o e n t r i e s 0 0 0 0 0 0 0 0 0 4 5 0 -3 P ¡ Q = i n e a c h r o w o f A W h a t a b o u t s p a r s e ( e . g . B o W ) h i s t o g r a m s ? T i m e C o m p l e x i t y : O ( S K )
The Earth Mover’s Distance
The Earth Mover’s Distance The Earth Mover’s Distance is defined as the  minimal cost  that must be paid to transform one histogram into the other, where there is a “ ground distance ” between the basic features that are aggregated into the histogram.
The Earth Mover’s Distance ≠
The Earth Mover’s Distance =
The Earth Mover’s Distance E M D D ( P ; Q ) = m i n F = f F i j g X i ; j F i j D i j s : t : X j F i j = P i X i F i j = Q j X i ; j F i j = X i P i = X j Q j = 1 F i j ¸ 0
The Earth Mover’s Distance E M D D ( P ; Q ) = m i n F = f F i j g P i ; j F i j D i j P i F i j s : t : X j F i j · P i X i F i j · Q j X i ; j F i j = m i n ( X i P i ; X j Q j ) f a l s e F i j ¸ 0
Pele and Werman 08 –  , a new EMD definition. The Earth Mover’s Distance \ E M D
Definition: \ E M D D C ( P ; Q ) = m i n F = f F i j g X i ; j F i j D i j + ¯ ¯ ¯ ¯ ¯ ¯ X i P i ¡ X j Q j ¯ ¯ ¯ ¯ ¯ ¯ £ C s : t : X j F i j · P i X i F i j · Q j X i ; j F i j = m i n ( X i P i ; X j Q j ) F i j ¸ 0 f a l s e
S u p p l i e r E M D D e m a n d e r D i j = 0 P D e m a n d e r · P S u p p l i e r
D e m a n d e r S u p p l i e r P D e m a n d e r · P S u p p l i e r D i j = C \ E M D
When to Use  When the total mass of two histograms is important.  E M D ¡ ; ¢ = E M D ¡ ; ¢ \ E M D
When to Use  When the total mass of two histograms is important.  ¡ ; ¢ < ¡ ; ¢ E M D ¡ ; ¢ = E M D ¡ ; ¢ \ E M D \ E M D \ E M D
When to Use  When the difference in total mass between histograms is a distinctive cue. E M D ¡ ; ¢ = E M D ¡ ; ¢ = 0 \ E M D
When to Use  When the difference in total mass between histograms is a distinctive cue. ¡ ; ¢ < ¡ ; ¢ E M D ¡ ; ¢ = E M D ¡ ; ¢ = 0 \ E M D \ E M D \ E M D
When to Use  If ground distance is a metric: ² E M D i s a m e t r i c o n l y f o r n o r m a l i z e d h i s t o g r a m s . ² \ E M D i s a m e t r i c f o r a l l h i s t o g r a m s ( C ¸ 1 2 ¢ ) . \ E M D
C = 1 D i j = ( 0 i f i = j 2 o t h e r w i s e \ E M D D C ( P ; Q ) = m i n F = f F i j g X i ; j F i j D i j + ¯ ¯ ¯ ¯ ¯ ¯ X i P i ¡ X j Q j ¯ ¯ ¯ ¯ ¯ ¯ £ C ² \ E M D = L 1 i f : \ E M D - a N a t u r a l E x t e n s i o n t o L 1
The Earth Mover’s Distance Complexity Zoo General ground distance: Orlin 88 normalized 1D histograms Werman, Peleg and Rosenfeld 85 L 1 O ( N 3 l o g N ) O ( N )
The Earth Mover’s Distance Complexity Zoo normalized 1D cyclic histograms Pele and Werman 08 (Werman, Peleg, Melter, and Kong 86) L 1 O ( N )
The Earth Mover’s Distance Complexity Zoo Manhattan grids Ling and Okada 07 O ( N 2 l o g N ( D + l o g N ) ) L 1
What about N-dimensional histograms with a cyclic dimensions? The Earth Mover’s Distance Complexity Zoo
What about N-dimensional histograms with a cyclic dimensions? The Earth Mover’s Distance Complexity Zoo
What about N-dimensional histograms with a cyclic dimensions? The Earth Mover’s Distance Complexity Zoo
The Earth Mover’s Distance Complexity Zoo general histograms Gudmundsson, Klein, Knauer and Smid 07 L 1 p 1 p 2 p 3 p 4 p 5 p 6 p 7 p 8 l 2 l 1 l ` 2 O ( N 2 l o g 2 D ¡ 1 N )
The Earth Mover’s Distance Complexity Zoo 1D linear/cyclic  histograms Pele and Werman 08 m i n ( L 1 ; 2 ) O ( N ) f 1 ; 2 ; : : : ; ¢ g
The Earth Mover’s Distance Complexity Zoo 1D linear/cyclic  histograms Pele and Werman 08 m i n ( L 1 ; 2 ) O ( N ) f 1 ; 2 ; : : : ; ¢ g
The Earth Mover’s Distance Complexity Zoo general  histograms is the number of edges with cost 1 Pele and Werman 08 m i n ( L 1 ; 2 ) f 1 ; 2 ; : : : ; ¢ g D K O ( N 2 K l o g ( N K ) )
The Earth Mover’s Distance Complexity Zoo Any thresholded distance  Pele and Werman 09 O ( N 2 l o g N ( K + l o g N ) ) number of edges with cost  different from the threshold K = O ( l o g N ) O ( N 2 l o g 2 N )
Thresholded Distances EMD with a thresholded ground distance is  not   an approximation of EMD. It has better performance.
The Flow Network Transformation Original Network Simplified Network
The Flow Network Transformation Original Network Simplified Network
The Flow Network Transformation Flowing the Monge sequence  (if ground distance is a metric,  zero-cost edges are a Monge sequence)
The Flow Network Transformation Removing Empty Bins and their edges
The Flow Network Transformation We actually finished here….
Combining Algorithms EMD algorithms can be combined. For example  : L 1
Combining Algorithms EMD algorithms can be combined. For example,  thresholded   : L 1
The Earth Mover’s Distance Approximations
Charikar 02, Indyk and Thaper 03 – approximated EMD on  by embedding it into the  norm. Time complexity:  Distortion (in expectation): The Earth Mover’s Distance Approximations L 1 f 1 ; : : : ; ¢ g d O ( T N d l o g ¢ ) O ( d l o g ¢ )
The Earth Mover’s Distance Approximations Grauman and Darrell 05 – Pyramid Match Kernel (PMK) same as Indyk and Thaper, replacing  with  histogram intersection . PMK approximates EMD with partial matching. PMK is a mercer kernel. Time complexity & distortion – same as Indyk and Thaper (proved in Grauman and Darrell 07).  L 1
The Earth Mover’s Distance Approximations Lazebnik, Schmid and Ponce 06 – used PMK in the spatial domain (SPM).
The Earth Mover’s Distance Approximations level 0 + + + + + + + + + + + + + + + + + + + + + + + + x 1 / 8 x 1 / 4 x 1 / 2 + + + level 1 level 2 + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
The Earth Mover’s Distance Approximations
The Earth Mover’s Distance Approximations Shirdhonkar and Jacobs 08 - approximated EMD  using the sum of absolute values of the weighted wavelet coefficients of the difference histogram. D i f f e r e n c e P Q Wavelet Transform j=0 j=1 Absolute values + £ 2 ¡ 2 £ 0 £ 2 ¡ 2 £ 1
The Earth Mover’s Distance Approximations Khot and Naor 06 – any embedding of the EMD over the d-dimensional Hamming cube into  must incur a distortion of  . Andoni, Indyk and Krauthgamer 08 - for sets with cardinalities upper bounded by a parameter  the distortion reduces to  . Naor and Schechtman  07 -  any embedding of the EMD over  must incur a distortion of  .  L 1 ­ ( d ) s O ( l o g s l o g d ) f 0 ; 1 ; : : : ; ¢ g 2 ­ ( p l o g ¢ )
Robust Distances Very   high distances  outliers   same difference.
Robust Distances With colors, the natural choice. 0 0 20 40 60 80 100 120 CIEDE200 Distance from Blue
Robust Distances ( b l u e ; r e d ) = 5 6 ¢E 0 0 ( b l u e ; y e l l o w ) = 1 0 2 ¢E 0 0
Robust Distances - Exponent Usually a negative exponent is used: ² L e t d ( a ; b ) b e a d i s t a n c e m e a s u r e b e t w e e n t w o f e a t u r e s - a ; b . ² T h e n e g a t i v e e x p o n e n t d i s t a n c e i s : d e ( a ; b ) = 1 ¡ e ¡ d ( a ; b ) ¾
Robust Distances - Exponent Exponent is used because  (Ruzon and Tomasi 01) : robust, smooth, monotonic, and a metric Input is always discrete anyway …
Robust Distances - Thresholded ² L e t d ( a ; b ) b e a d i s t a n c e m e a s u r e b e t w e e n t w o f e a t u r e s - a ; b . ² T h e t h r e s h o l d e d d i s t a n c e w i t h a t h r e s h o l d o f t > 0 i s : d t ( a ; b ) = m i n ( d ( a ; b ) ; t ) .
Thresholded Distances Thresholded metrics are also metrics (Pele and Werman ICCV 2009). Better results. Pele and Werman ICCV 2009 algorithm computes EMD with thresholded ground distances much faster. Thresholded distance corresponds to  sparse similarities matrix  -> faster QC / QF computation. A i j = 1 ¡ D i j m a x i j ( D i j )
Thresholded Distances Thresholded  vs.  exponent : Fast computation of cross-bin distances with a  thresholded  ground distance. Exponent  changes small distances – can be a problem (e.g. color differences).
Thresholded Distances Color distance should be thresholded (robust).  0 50 100 150 distance from blue ¢ E 0 0 m i n ( ¢ E 0 0 ; 1 0 ) 1 0 ( 1 ¡ e ¡ ¢ E 0 0 4 ) 1 0 ( 1 ¡ e ¡ ¢ E 0 0 5 )
Thresholded Distances Exponent changes small distances ¢ E 0 0 m i n ( ¢ E 0 0 ; 1 0 ) 1 0 ( 1 ¡ e ¡ ¢ E 0 0 4 ) 1 0 ( 1 ¡ e ¡ ¢ E 0 0 5 ) 0 5 10 distance from blue
A Ground Distance for SIFT  d R = j j ( x i ; y i ) ¡ ( x j ; y j ) j j 2 + m i n ( j o i ¡ o j j ; M ¡ j o i ¡ o j j ) d T = m i n ( d R ; T ) T h e g r o u n d d i s t a n c e b e t w e e n t w o S I F T b i n s ( x i ; y i ; o i ) a n d ( x j ; y j ; o j ) :
A Ground Distance for Color Image T h e g r o u n d d i s t a n c e s b e t w e e n t w o L A B i m a g e b i n s ( x i ; y i ; L i ; a i ; b i ) a n d ( x j ; y j ; L j ; a j ; b j ) w e u s e a r e : d c T = m i n ( ( j j ( x i ; y i ) ¡ ( x j ; y j ) j j 2 ) + ¢ 0 0 ( ( L i ; a i ; b i ) ; ( L j ; a j ; b j ) ) ; T )
Perceptual Color Differences
Perceptual Color Differences Euclidean distance on L*a*b* space is widely considered as perceptual uniform. 0 100 200 300 distance from blue j j L a b j j 2
Perceptual Color Differences 0 20 40 60 adist2 distance from blue Purples before blues j j L a b j j 2
Perceptual Color Differences on L*a*b* space is better. Luo, Cui and Rigg 01. Sharma, Wu and Dalal 05.  distance from blue 0 5 10 15 20 ¢ E 0 0 ¢ E 0 0
Perceptual Color Differences on L*a*b* space is better. ¢ E 0 0 j j L a b j j 2 ¢ E 0 0
Perceptual Color Differences 0 50 100 150 distance from blue on L*a*b* space is better. ¢ E 0 0 ¢ E 0 0
Perceptual Color Differences on L*a*b* space is better. But still has major problems. Color distance should be thresholded (robust).  ( b l u e ; r e d ) = 5 6 ¢E 0 0 ( b l u e ; y e l l o w ) = 1 0 2 ¢E 0 0 ¢ E 0 0
Perceptual Color Differences Color distance should be saturated (robust).  0 50 100 150 distance from blue ¢ E 0 0 m i n ( ¢ E 0 0 ; 1 0 ) 1 0 ( 1 ¡ e ¡ ¢ E 0 0 4 ) 1 0 ( 1 ¡ e ¡ ¢ E 0 0 5 )
Thresholded Distances Exponent changes small distances ¢ E 0 0 m i n ( ¢ E 0 0 ; 1 0 ) 1 0 ( 1 ¡ e ¡ ¢ E 0 0 4 ) 1 0 ( 1 ¡ e ¡ ¢ E 0 0 5 ) 0 5 10 distance from blue
Perceptual Color Descriptors
Perceptual Color Descriptors 11 basic color terms.  Berlin and Kay 69. white red green yellow blue brown purple pink orange grey black
Perceptual Color Descriptors 11 basic color terms.  Berlin and Kay 69.
Perceptual Color Descriptors 11 basic color terms.  Berlin and Kay 69. Image copyright by Eric Rolph. Taken from: http://upload.wikimedia.org/wikipedia/commons/5/5c/Double-alaskan-rainbow.jpg
Perceptual Color Descriptors 11 basic color terms.  Berlin and Kay 69.
Perceptual Color Descriptors How to give each pixel an “11-colors” description ?
Perceptual Color Descriptors Learning Color Names from Real-World Images,  J. van de Weijer, C. Schmid, J. Verbeek CVPR 2007.
Perceptual Color Descriptors Learning Color Names from Real-World Images,  J. van de Weijer, C. Schmid, J. Verbeek CVPR 2007.
Perceptual Color Descriptors Learning Color Names from Real-World Images,  J. van de Weijer, C. Schmid, J. Verbeek CVPR 2007.  chip-based real-world
Perceptual Color Descriptors Learning Color Names from Real-World Images,  J. van de Weijer, C. Schmid, J. Verbeek CVPR 2007.  For each color returns a probability distribution over the 11 basic colors. white red green yellow blue brown purple pink orange grey black
Perceptual Color Descriptors Applying Color Names to Image Description,  J. van de Weijer, C. Schmid ICIP 2007. Outperformed state of the art color descriptors.
Perceptual Color Descriptors Using illumination invariants – black, gray and white are the same.  “ Too much invariance” happens in other cases  ( Local features and kernels for classification of texture and object categories: An in-depth study - Zhang, Marszalek, Lazebnik and Schmid. IJCV 2007, Learning the discriminative power-invariance trade-off - Varma and Ray. ICCV 2007 ). To conclude: Don’t solve imaginary problems.
Perceptual Color Descriptors This method is still not perfect. 11 color vector for  purple (255,0,255)   is: In real world images there are no such  over-saturated colors. 0 0 0 0 0 0 0 0 0 1 0
Open Questions EMD variant that reduces the effect of large bins. Learning the ground distance for EMD. Learning the similarity matrix and normalization factor for QC.
Hands-On Code Example http://www.cs.huji.ac.il/~ofirpele/FastEMD/code/ http://www.cs.huji.ac.il/~ofirpele/QC/code/
Tutorial: Or  “Ofir Pele” http://www.cs.huji.ac.il/~ofirpele/DFML_ECCV2010_tutorial/

More Related Content

What's hot (20)

PPTX
18 polar coordinates x
math260
 
PDF
Single source shortes path in dag
Kiran K
 
PPTX
Linear Combination, Span And Linearly Independent, Dependent Set
Dhaval Shukla
 
PDF
String Matching with Finite Automata and Knuth Morris Pratt Algorithm
Kiran K
 
PDF
Bellman ford
Kiran K
 
PDF
Longest common subsequence
Kiran K
 
PPT
1 polar coordinates
math267
 
PDF
Qp cdsi18-math
PRASANTH RAKOTI
 
DOCX
Polar coordinates
Tarun Gehlot
 
PPTX
20 polar equations and graphs
math267
 
PPTX
35 tangent and arc length in polar coordinates
math266
 
PDF
Pc 10.7 notes_polar
Jonathan Fjelstrom
 
PDF
Diktat Geometri Transformasi (English)
Fajar Arwadi
 
PPTX
Transformación de coordenadas
Andres32698
 
PPT
Introduction to coordinate geometry by pratima nayak
Pratima Nayak ,Kendriya Vidyalaya Sangathan
 
PPTX
33 parametric equations x
math266
 
PDF
BEADS : filtrage asymétrique de ligne de base (tendance) et débruitage pour d...
Laurent Duval
 
PPTX
36 area in polar coordinate
math266
 
PDF
Dokumen.tips mathematics ii-institute-of-aeronautical-engineering-pptpdfadvan...
Mahmood Adel
 
PPTX
34 polar coordinate and equations
math266
 
18 polar coordinates x
math260
 
Single source shortes path in dag
Kiran K
 
Linear Combination, Span And Linearly Independent, Dependent Set
Dhaval Shukla
 
String Matching with Finite Automata and Knuth Morris Pratt Algorithm
Kiran K
 
Bellman ford
Kiran K
 
Longest common subsequence
Kiran K
 
1 polar coordinates
math267
 
Qp cdsi18-math
PRASANTH RAKOTI
 
Polar coordinates
Tarun Gehlot
 
20 polar equations and graphs
math267
 
35 tangent and arc length in polar coordinates
math266
 
Pc 10.7 notes_polar
Jonathan Fjelstrom
 
Diktat Geometri Transformasi (English)
Fajar Arwadi
 
Transformación de coordenadas
Andres32698
 
Introduction to coordinate geometry by pratima nayak
Pratima Nayak ,Kendriya Vidyalaya Sangathan
 
33 parametric equations x
math266
 
BEADS : filtrage asymétrique de ligne de base (tendance) et débruitage pour d...
Laurent Duval
 
36 area in polar coordinate
math266
 
Dokumen.tips mathematics ii-institute-of-aeronautical-engineering-pptpdfadvan...
Mahmood Adel
 
34 polar coordinate and equations
math266
 

Similar to ECCV2010: distance function and metric learning part 1 (20)

PPTX
Clasification approaches
gscprasad1111
 
PDF
Google BigQuery is a very popular enterprise warehouse that’s built with a co...
Abebe Admasu
 
PPTX
Different kind of distance and Statistical Distance
Khulna University
 
PPTX
Data Mining Lecture_5.pptx
Subrata Kumer Paul
 
PPT
Lecture slides week14-15
Shani729
 
PDF
ECCV2010: distance function and metric learning part 2
zukun
 
PDF
Cluster Analysis: Measuring Similarity & Dissimilarity
ShivarkarSandip
 
PPTX
Distance function
Lippo Group Digital
 
PPT
Cs345 cl
mattriley
 
PDF
Pattern learning and recognition on statistical manifolds: An information-geo...
Frank Nielsen
 
PPTX
komal (distance and similarity measure).pptx
lovepreet33653
 
PDF
Clustering Algorithms - Kmeans,Min ALgorithm
Sharmila Chidaravalli
 
PPT
Ds2 statistics
ShousanLiao
 
PDF
A COMPARATIVE STUDY ON DISTANCE MEASURING APPROACHES FOR CLUSTERING
IJORCS
 
PPTX
SkNoushadddoja_28100119039.pptx
PrakasBhowmik
 
PDF
09Evaluation_Clustering.pdf
BizuayehuDesalegn
 
PPT
similarities-knn-1.ppt
satvikpatil5
 
PDF
Cluster analysis
Hohai university
 
PPTX
Data mining Measuring similarity and desimilarity
Rushali Deshmukh
 
PDF
Neighborhood Component Analysis 20071108
Ting-Shuo Yo
 
Clasification approaches
gscprasad1111
 
Google BigQuery is a very popular enterprise warehouse that’s built with a co...
Abebe Admasu
 
Different kind of distance and Statistical Distance
Khulna University
 
Data Mining Lecture_5.pptx
Subrata Kumer Paul
 
Lecture slides week14-15
Shani729
 
ECCV2010: distance function and metric learning part 2
zukun
 
Cluster Analysis: Measuring Similarity & Dissimilarity
ShivarkarSandip
 
Distance function
Lippo Group Digital
 
Cs345 cl
mattriley
 
Pattern learning and recognition on statistical manifolds: An information-geo...
Frank Nielsen
 
komal (distance and similarity measure).pptx
lovepreet33653
 
Clustering Algorithms - Kmeans,Min ALgorithm
Sharmila Chidaravalli
 
Ds2 statistics
ShousanLiao
 
A COMPARATIVE STUDY ON DISTANCE MEASURING APPROACHES FOR CLUSTERING
IJORCS
 
SkNoushadddoja_28100119039.pptx
PrakasBhowmik
 
09Evaluation_Clustering.pdf
BizuayehuDesalegn
 
similarities-knn-1.ppt
satvikpatil5
 
Cluster analysis
Hohai university
 
Data mining Measuring similarity and desimilarity
Rushali Deshmukh
 
Neighborhood Component Analysis 20071108
Ting-Shuo Yo
 
Ad

More from zukun (20)

PDF
My lyn tutorial 2009
zukun
 
PDF
ETHZ CV2012: Tutorial openCV
zukun
 
PDF
ETHZ CV2012: Information
zukun
 
PDF
Siwei lyu: natural image statistics
zukun
 
PDF
Lecture9 camera calibration
zukun
 
PDF
Brunelli 2008: template matching techniques in computer vision
zukun
 
PDF
Modern features-part-4-evaluation
zukun
 
PDF
Modern features-part-3-software
zukun
 
PDF
Modern features-part-2-descriptors
zukun
 
PDF
Modern features-part-1-detectors
zukun
 
PDF
Modern features-part-0-intro
zukun
 
PDF
Lecture 02 internet video search
zukun
 
PDF
Lecture 01 internet video search
zukun
 
PDF
Lecture 03 internet video search
zukun
 
PDF
Icml2012 tutorial representation_learning
zukun
 
PPT
Advances in discrete energy minimisation for computer vision
zukun
 
PDF
Gephi tutorial: quick start
zukun
 
PDF
EM algorithm and its application in probabilistic latent semantic analysis
zukun
 
PDF
Object recognition with pictorial structures
zukun
 
PDF
Iccv2011 learning spatiotemporal graphs of human activities
zukun
 
My lyn tutorial 2009
zukun
 
ETHZ CV2012: Tutorial openCV
zukun
 
ETHZ CV2012: Information
zukun
 
Siwei lyu: natural image statistics
zukun
 
Lecture9 camera calibration
zukun
 
Brunelli 2008: template matching techniques in computer vision
zukun
 
Modern features-part-4-evaluation
zukun
 
Modern features-part-3-software
zukun
 
Modern features-part-2-descriptors
zukun
 
Modern features-part-1-detectors
zukun
 
Modern features-part-0-intro
zukun
 
Lecture 02 internet video search
zukun
 
Lecture 01 internet video search
zukun
 
Lecture 03 internet video search
zukun
 
Icml2012 tutorial representation_learning
zukun
 
Advances in discrete energy minimisation for computer vision
zukun
 
Gephi tutorial: quick start
zukun
 
EM algorithm and its application in probabilistic latent semantic analysis
zukun
 
Object recognition with pictorial structures
zukun
 
Iccv2011 learning spatiotemporal graphs of human activities
zukun
 
Ad

Recently uploaded (20)

PPTX
How to Close Subscription in Odoo 18 - Odoo Slides
Celine George
 
PPTX
Artificial Intelligence in Gastroentrology: Advancements and Future Presprec...
AyanHossain
 
PPTX
Unlock the Power of Cursor AI: MuleSoft Integrations
Veera Pallapu
 
PPTX
Translation_ Definition, Scope & Historical Development.pptx
DhatriParmar
 
PDF
The-Invisible-Living-World-Beyond-Our-Naked-Eye chapter 2.pdf/8th science cur...
Sandeep Swamy
 
PPTX
20250924 Navigating the Future: How to tell the difference between an emergen...
McGuinness Institute
 
PPTX
CONCEPT OF CHILD CARE. pptx
AneetaSharma15
 
PPTX
Python-Application-in-Drug-Design by R D Jawarkar.pptx
Rahul Jawarkar
 
PPTX
Digital Professionalism and Interpersonal Competence
rutvikgediya1
 
PPTX
LDP-2 UNIT 4 Presentation for practical.pptx
abhaypanchal2525
 
PPTX
Applications of matrices In Real Life_20250724_091307_0000.pptx
gehlotkrish03
 
PDF
BÀI TẬP TEST BỔ TRỢ THEO TỪNG CHỦ ĐỀ CỦA TỪNG UNIT KÈM BÀI TẬP NGHE - TIẾNG A...
Nguyen Thanh Tu Collection
 
DOCX
Modul Ajar Deep Learning Bahasa Inggris Kelas 11 Terbaru 2025
wahyurestu63
 
PPTX
Gupta Art & Architecture Temple and Sculptures.pptx
Virag Sontakke
 
PDF
My Thoughts On Q&A- A Novel By Vikas Swarup
Niharika
 
DOCX
Unit 5: Speech-language and swallowing disorders
JELLA VISHNU DURGA PRASAD
 
PPTX
K-Circle-Weekly-Quiz12121212-May2025.pptx
Pankaj Rodey
 
PPT
DRUGS USED IN THERAPY OF SHOCK, Shock Therapy, Treatment or management of shock
Rajshri Ghogare
 
PPTX
How to Track Skills & Contracts Using Odoo 18 Employee
Celine George
 
PPTX
Basics and rules of probability with real-life uses
ravatkaran694
 
How to Close Subscription in Odoo 18 - Odoo Slides
Celine George
 
Artificial Intelligence in Gastroentrology: Advancements and Future Presprec...
AyanHossain
 
Unlock the Power of Cursor AI: MuleSoft Integrations
Veera Pallapu
 
Translation_ Definition, Scope & Historical Development.pptx
DhatriParmar
 
The-Invisible-Living-World-Beyond-Our-Naked-Eye chapter 2.pdf/8th science cur...
Sandeep Swamy
 
20250924 Navigating the Future: How to tell the difference between an emergen...
McGuinness Institute
 
CONCEPT OF CHILD CARE. pptx
AneetaSharma15
 
Python-Application-in-Drug-Design by R D Jawarkar.pptx
Rahul Jawarkar
 
Digital Professionalism and Interpersonal Competence
rutvikgediya1
 
LDP-2 UNIT 4 Presentation for practical.pptx
abhaypanchal2525
 
Applications of matrices In Real Life_20250724_091307_0000.pptx
gehlotkrish03
 
BÀI TẬP TEST BỔ TRỢ THEO TỪNG CHỦ ĐỀ CỦA TỪNG UNIT KÈM BÀI TẬP NGHE - TIẾNG A...
Nguyen Thanh Tu Collection
 
Modul Ajar Deep Learning Bahasa Inggris Kelas 11 Terbaru 2025
wahyurestu63
 
Gupta Art & Architecture Temple and Sculptures.pptx
Virag Sontakke
 
My Thoughts On Q&A- A Novel By Vikas Swarup
Niharika
 
Unit 5: Speech-language and swallowing disorders
JELLA VISHNU DURGA PRASAD
 
K-Circle-Weekly-Quiz12121212-May2025.pptx
Pankaj Rodey
 
DRUGS USED IN THERAPY OF SHOCK, Shock Therapy, Treatment or management of shock
Rajshri Ghogare
 
How to Track Skills & Contracts Using Odoo 18 Employee
Celine George
 
Basics and rules of probability with real-life uses
ravatkaran694
 

ECCV2010: distance function and metric learning part 1

  • 1. Distance Functions and Metric Learning: Part 1 12/1/2011 version. Updated Version: http://www.cs.huji.ac.il/~ofirpele/DFML_ECCV2010_tutorial/ Michael Werman Ofir Pele The Hebrew University of Jerusalem
  • 2. Rough Outline Bin-to-Bin Distances. Cross-Bin Distances: Quadratic-Form (aka Mahalanobis) / Quadratic-Chi. The Earth Mover’s Distance. Perceptual Color Differences. Hands-On Code Example.
  • 4. Metric ² N o n - n e g a t i v i t y : D ( P ; Q ) ¸ 0 ² I d e n t i t y o f i n d i s c e r n i b l e s : D ( P ; Q ) = 0 i ® P = Q ² S y m m e t r y : D ( P ; Q ) = D ( Q ; P ) ² S u b a d d i t i v i t y ( t r i a n g l e i n e q u a l i t y ) : D ( P ; Q ) · D ( P ; K ) + D ( K ; Q )
  • 5. Pseudo-Metric (aka Semi-Metric) ² N o n - n e g a t i v i t y : D ( P ; Q ) ¸ 0 ² P r o p e r t y c h a n g e d t o : D ( P ; Q ) = 0 i f P = Q ² S y m m e t r y : D ( P ; Q ) = D ( Q ; P ) ² S u b a d d i t i v i t y ( t r i a n g l e i n e q u a l i t y ) : D ( P ; Q ) · D ( P ; K ) + D ( K ; Q )
  • 6. Metric A n o b j e c t i s m o s t s i m i l a r t o i t s e l f ² N o n - n e g a t i v i t y : D ( P ; Q ) ¸ 0 ² I d e n t i t y o f i n d i s c e r n i b l e s : D ( P ; Q ) = 0 i ® P = Q
  • 7. Metric ² S y m m e t r y : D ( P ; Q ) = D ( Q ; P ) ² S u b a d d i t i v i t y ( t r i a n g l e i n e q u a l i t y ) : D ( P ; Q ) · D ( P ; K ) + D ( K ; Q ) U s e f u l f o r m a n y a l g o r i t h m s
  • 8. Minkowski-Form Distances L p ( P ; Q ) = Ã X i j P i ¡ Q i j p ! 1 p
  • 9. Minkowski-Form Distances L 2 ( P ; Q ) = s X i ( P i ¡ Q i ) 2 L 1 ( P ; Q ) = X i j P i ¡ Q i j L 1 ( P ; Q ) = m a x i j P i ¡ Q i j
  • 10. Kullback-Leibler Divergence Information theoretic origin. Non symmetric. K L ( P ; Q ) = X i P i l o g P i Q i Q i = 0 ?
  • 11. Jensen-Shannon Divergence J S ( P ; Q ) = 1 2 K L ( P ; M ) + 1 2 K L ( Q ; M ) M = 1 2 ( P + Q ) J S ( P ; Q ) = 1 2 X i P i l o g 2 P i P i + Q i + 1 2 X i Q i l o g 2 Q i P i + Q i
  • 12. Jensen-Shannon Divergence Information Theoretic origin. Symmetric. is a metric. J S ( P ; Q ) = 1 2 K L ( P ; M ) + 1 2 K L ( Q ; M ) M = 1 2 ( P + Q ) p J S
  • 13. Jensen-Shannon Divergence Using Taylor extension and some algebra: J S ( P ; Q ) = 1 X n = 1 1 2 n ( 2 n ¡ 1 ) X i ( P i ¡ Q i ) 2 n ( P i + Q i ) 2 n ¡ 1 = 1 2 X i ( P i ¡ Q i ) 2 ( P i + Q i ) + 1 1 2 X i ( P i ¡ Q i ) 4 ( P i + Q i ) 3 + : : :
  • 14. Histogram Distance Statistical origin. Experimentally results are very similar to JS. Reduces the effect of large bins. is a metric  2  2 ( P ; Q ) = 1 2 X i ( P i ¡ Q i ) 2 ( P i + Q i ) p  2
  • 15. Histogram Distance < Â 2 Â 2 ( ; ) Â 2 ( ; )
  • 16. Histogram Distance > Â 2 L 1 ( ; ) L 1 ( ; )
  • 17. Histogram Distance Experimentally better than . Â 2 L 2 Â 2 ( P ; Q ) = 1 2 X i ( P i ¡ Q i ) 2 ( P i + Q i )
  • 18. Bin-to-Bin Distances Bin-to-Bin distances such as L2 are sensitive to quantization: > L 1 ( ; ) L 1 ; L 2 ; Â 2 L 1 ( ; )
  • 19. #bins #bins Bin-to-Bin Distances robustness distinctiveness robustness distinctiveness > L 1 ( ; ) L 1 ( ; )
  • 20. #bins #bins Bin-to-Bin Distances robustness distinctiveness robustness distinctiveness Can we achieve robustness and distinctiveness ?
  • 21. The Quadratic-Form Histogram Distance is the similarity between bin i and j. If is the inverse of the covariance matrix, QF is called Mahalanobis distance. A i j = s X i j ( P i ¡ Q i ) ( P j ¡ Q j ) A i j ( 1 ) Q F A ( P ; Q ) = q ( P ¡ Q ) T A ( P ¡ Q ) ( 1 ) A
  • 22. The Quadratic-Form Histogram Distance Q F A ( P ; Q ) = q ( P ¡ Q ) T A ( P ¡ Q ) = s X i j ( P i ¡ Q i ) ( P j ¡ Q j ) A i j A = I = s X i j ( P i ¡ Q i ) 2 = L 2 ( P ; Q )
  • 23. The Quadratic-Form Histogram Distance Does not reduce the effect of large bins. Alleviates the quantization problem. Linear time computation in # non zero . A i j Q F A ( P ; Q ) = q ( P ¡ Q ) T A ( P ¡ Q )
  • 24. The Quadratic-Form Histogram Distance If A is positive-semidefinite then QF is a pseudo-metric. Q F A ( P ; Q ) = q ( P ¡ Q ) T A ( P ¡ Q )
  • 25. Pseudo-Metric (aka Semi-Metric) ² N o n - n e g a t i v i t y : D ( P ; Q ) ¸ 0 ² P r o p e r t y c h a n g e d t o : D ( P ; Q ) = 0 i f P = Q ² S y m m e t r y : D ( P ; Q ) = D ( Q ; P ) ² S u b a d d i t i v i t y ( t r i a n g l e i n e q u a l i t y ) : D ( P ; Q ) · D ( P ; K ) + D ( K ; Q )
  • 26. The Quadratic-Form Histogram Distance If A is positive-definite then QF is a metric. Q F A ( P ; Q ) = q ( P ¡ Q ) T A ( P ¡ Q )
  • 27. The Quadratic-Form Histogram Distance Q F A ( P ; Q ) = q ( P ¡ Q ) T A ( P ¡ Q ) = q ( P ¡ Q ) T W T W ( P ¡ Q ) = L 2 ( W P ; W Q )
  • 28. The Quadratic-Form Histogram Distance We assume there is a linear transformation that makes bins independent There are cases where this is not true e.g. COLOR Q F A ( P ; Q ) = q ( P ¡ Q ) T A ( P ¡ Q ) = L 2 ( W P ; W Q )
  • 29. The Quadratic-Form Histogram Distance Converting distance to similarity (Hafner et. al 95): Q F A ( P ; Q ) = q ( P ¡ Q ) T A ( P ¡ Q ) A i j = 1 ¡ D i j m a x i j ( D i j )
  • 30. The Quadratic-Form Histogram Distance Converting distance to similarity (Hafner et. al 95): Q F A ( P ; Q ) = q ( P ¡ Q ) T A ( P ¡ Q ) A i j = e ¡ ® D ( i ; j ) m a x i j ( D ( i ; j ) ) I f ® i s l a r g e e n o u g h , A w i l l b e p o s i t i v e - d e ¯ n i t i v e
  • 31. The Quadratic-Form Histogram Distance Learning the similarity matrix: part 2 of this tutorial. Q F A ( P ; Q ) = q ( P ¡ Q ) T A ( P ¡ Q )
  • 32. The Quadratic-Chi Histogram Distance is the similarity between bin i and j. Generalizes and . Reduces the effect of large bins. Alleviates the quantization problem. Linear time computation in # non zero . Â 2 Q C A m ( P ; Q ) = v u u t X i j ( P i ¡ Q i ) ( P j ¡ Q j ) A i j ( P c ( P c + Q c ) A c i ) m ( P c ( P c + Q c ) A c j ) m A i j A i j Q F
  • 33. The Quadratic-Chi Histogram Distance is non-negative if A is positive-semidefinite. Symmetric. Triangle inequality unknown. If we define and , QC is continuous. Q C A m ( P ; Q ) = v u u t X i j ( P i ¡ Q i ) ( P j ¡ Q j ) A i j ( P c ( P c + Q c ) A c i ) m ( P c ( P c + Q c ) A c j ) m 0 · m < 1 0 0 = 0
  • 35. Sparseness-Invariant = D ) ( , Empty bins D ( , )
  • 36. The Quadratic-Chi Histogram Distance Code f u n c t i o n d i s t = Q u a d r a t i c C h i ( P , Q , A , m ) Z = ( P + Q ) * A ; % 1 c a n b e a n y n u m b e r a s Z _ i = = 0 i f f D _ i = 0 Z ( Z = = 0 ) = 1 ; Z = Z . ^ m ; D = ( P - Q ) . / Z ; % m a x i s r e d u n d a n t i f A i s % p o s i t i v e - s e m i d e f i n i t e d i s t = s q r t ( m a x ( D * A * D ' , 0 ) ) ;
  • 37. The Quadratic-Chi Histogram Distance Code f u n c t i o n d i s t = Q u a d r a t i c C h i ( P , Q , A , m ) Z = ( P + Q ) * A ; % 1 c a n b e a n y n u m b e r a s Z _ i = = 0 i f f D _ i = 0 Z ( Z = = 0 ) = 1 ; Z = Z . ^ m ; D = ( P - Q ) . / Z ; % m a x i s r e d u n d a n t i f A i s % p o s i t i v e - s e m i d e f i n i t e d i s t = s q r t ( m a x ( D * A * D ' , 0 ) ) ; T i m e C o m p l e x i t y : O ( # A i j 6 = 0 )
  • 38. The Quadratic-Chi Histogram Distance Code f u n c t i o n d i s t = Q u a d r a t i c C h i ( P , Q , A , m ) Z = ( P + Q ) * A ; % 1 c a n b e a n y n u m b e r a s Z _ i = = 0 i f f D _ i = 0 Z ( Z = = 0 ) = 1 ; Z = Z . ^ m ; D = ( P - Q ) . / Z ; % m a x i s r e d u n d a n t i f A i s % p o s i t i v e - s e m i d e f i n i t e d i s t = s q r t ( m a x ( D * A * D ' , 0 ) ) ; W h a t a b o u t s p a r s e ( e . g . B o W ) h i s t o g r a m s ?
  • 39. The Quadratic-Chi Histogram Distance Code 0 0 0 0 0 0 0 0 0 4 5 0 -3 P ¡ Q = W h a t a b o u t s p a r s e ( e . g . B o W ) h i s t o g r a m s ? T i m e C o m p l e x i t y : O ( S K )
  • 40. The Quadratic-Chi Histogram Distance Code 0 0 0 0 0 0 0 0 0 4 5 0 -3 P ¡ Q = # ( P 6 = 0 ) + # ( Q 6 = 0 ) W h a t a b o u t s p a r s e ( e . g . B o W ) h i s t o g r a m s ? T i m e C o m p l e x i t y : O ( S K )
  • 41. The Quadratic-Chi Histogram Distance Code A v e r a g e o f n o n - z e r o e n t r i e s 0 0 0 0 0 0 0 0 0 4 5 0 -3 P ¡ Q = i n e a c h r o w o f A W h a t a b o u t s p a r s e ( e . g . B o W ) h i s t o g r a m s ? T i m e C o m p l e x i t y : O ( S K )
  • 43. The Earth Mover’s Distance The Earth Mover’s Distance is defined as the minimal cost that must be paid to transform one histogram into the other, where there is a “ ground distance ” between the basic features that are aggregated into the histogram.
  • 44. The Earth Mover’s Distance ≠
  • 45. The Earth Mover’s Distance =
  • 46. The Earth Mover’s Distance E M D D ( P ; Q ) = m i n F = f F i j g X i ; j F i j D i j s : t : X j F i j = P i X i F i j = Q j X i ; j F i j = X i P i = X j Q j = 1 F i j ¸ 0
  • 47. The Earth Mover’s Distance E M D D ( P ; Q ) = m i n F = f F i j g P i ; j F i j D i j P i F i j s : t : X j F i j · P i X i F i j · Q j X i ; j F i j = m i n ( X i P i ; X j Q j ) f a l s e F i j ¸ 0
  • 48. Pele and Werman 08 – , a new EMD definition. The Earth Mover’s Distance \ E M D
  • 49. Definition: \ E M D D C ( P ; Q ) = m i n F = f F i j g X i ; j F i j D i j + ¯ ¯ ¯ ¯ ¯ ¯ X i P i ¡ X j Q j ¯ ¯ ¯ ¯ ¯ ¯ £ C s : t : X j F i j · P i X i F i j · Q j X i ; j F i j = m i n ( X i P i ; X j Q j ) F i j ¸ 0 f a l s e
  • 50. S u p p l i e r E M D D e m a n d e r D i j = 0 P D e m a n d e r · P S u p p l i e r
  • 51. D e m a n d e r S u p p l i e r P D e m a n d e r · P S u p p l i e r D i j = C \ E M D
  • 52. When to Use When the total mass of two histograms is important. E M D ¡ ; ¢ = E M D ¡ ; ¢ \ E M D
  • 53. When to Use When the total mass of two histograms is important. ¡ ; ¢ < ¡ ; ¢ E M D ¡ ; ¢ = E M D ¡ ; ¢ \ E M D \ E M D \ E M D
  • 54. When to Use When the difference in total mass between histograms is a distinctive cue. E M D ¡ ; ¢ = E M D ¡ ; ¢ = 0 \ E M D
  • 55. When to Use When the difference in total mass between histograms is a distinctive cue. ¡ ; ¢ < ¡ ; ¢ E M D ¡ ; ¢ = E M D ¡ ; ¢ = 0 \ E M D \ E M D \ E M D
  • 56. When to Use If ground distance is a metric: ² E M D i s a m e t r i c o n l y f o r n o r m a l i z e d h i s t o g r a m s . ² \ E M D i s a m e t r i c f o r a l l h i s t o g r a m s ( C ¸ 1 2 ¢ ) . \ E M D
  • 57. C = 1 D i j = ( 0 i f i = j 2 o t h e r w i s e \ E M D D C ( P ; Q ) = m i n F = f F i j g X i ; j F i j D i j + ¯ ¯ ¯ ¯ ¯ ¯ X i P i ¡ X j Q j ¯ ¯ ¯ ¯ ¯ ¯ £ C ² \ E M D = L 1 i f : \ E M D - a N a t u r a l E x t e n s i o n t o L 1
  • 58. The Earth Mover’s Distance Complexity Zoo General ground distance: Orlin 88 normalized 1D histograms Werman, Peleg and Rosenfeld 85 L 1 O ( N 3 l o g N ) O ( N )
  • 59. The Earth Mover’s Distance Complexity Zoo normalized 1D cyclic histograms Pele and Werman 08 (Werman, Peleg, Melter, and Kong 86) L 1 O ( N )
  • 60. The Earth Mover’s Distance Complexity Zoo Manhattan grids Ling and Okada 07 O ( N 2 l o g N ( D + l o g N ) ) L 1
  • 61. What about N-dimensional histograms with a cyclic dimensions? The Earth Mover’s Distance Complexity Zoo
  • 62. What about N-dimensional histograms with a cyclic dimensions? The Earth Mover’s Distance Complexity Zoo
  • 63. What about N-dimensional histograms with a cyclic dimensions? The Earth Mover’s Distance Complexity Zoo
  • 64. The Earth Mover’s Distance Complexity Zoo general histograms Gudmundsson, Klein, Knauer and Smid 07 L 1 p 1 p 2 p 3 p 4 p 5 p 6 p 7 p 8 l 2 l 1 l ` 2 O ( N 2 l o g 2 D ¡ 1 N )
  • 65. The Earth Mover’s Distance Complexity Zoo 1D linear/cyclic histograms Pele and Werman 08 m i n ( L 1 ; 2 ) O ( N ) f 1 ; 2 ; : : : ; ¢ g
  • 66. The Earth Mover’s Distance Complexity Zoo 1D linear/cyclic histograms Pele and Werman 08 m i n ( L 1 ; 2 ) O ( N ) f 1 ; 2 ; : : : ; ¢ g
  • 67. The Earth Mover’s Distance Complexity Zoo general histograms is the number of edges with cost 1 Pele and Werman 08 m i n ( L 1 ; 2 ) f 1 ; 2 ; : : : ; ¢ g D K O ( N 2 K l o g ( N K ) )
  • 68. The Earth Mover’s Distance Complexity Zoo Any thresholded distance Pele and Werman 09 O ( N 2 l o g N ( K + l o g N ) ) number of edges with cost different from the threshold K = O ( l o g N ) O ( N 2 l o g 2 N )
  • 69. Thresholded Distances EMD with a thresholded ground distance is not an approximation of EMD. It has better performance.
  • 70. The Flow Network Transformation Original Network Simplified Network
  • 71. The Flow Network Transformation Original Network Simplified Network
  • 72. The Flow Network Transformation Flowing the Monge sequence (if ground distance is a metric, zero-cost edges are a Monge sequence)
  • 73. The Flow Network Transformation Removing Empty Bins and their edges
  • 74. The Flow Network Transformation We actually finished here….
  • 75. Combining Algorithms EMD algorithms can be combined. For example : L 1
  • 76. Combining Algorithms EMD algorithms can be combined. For example, thresholded : L 1
  • 77. The Earth Mover’s Distance Approximations
  • 78. Charikar 02, Indyk and Thaper 03 – approximated EMD on by embedding it into the norm. Time complexity: Distortion (in expectation): The Earth Mover’s Distance Approximations L 1 f 1 ; : : : ; ¢ g d O ( T N d l o g ¢ ) O ( d l o g ¢ )
  • 79. The Earth Mover’s Distance Approximations Grauman and Darrell 05 – Pyramid Match Kernel (PMK) same as Indyk and Thaper, replacing with histogram intersection . PMK approximates EMD with partial matching. PMK is a mercer kernel. Time complexity & distortion – same as Indyk and Thaper (proved in Grauman and Darrell 07). L 1
  • 80. The Earth Mover’s Distance Approximations Lazebnik, Schmid and Ponce 06 – used PMK in the spatial domain (SPM).
  • 81. The Earth Mover’s Distance Approximations level 0 + + + + + + + + + + + + + + + + + + + + + + + + x 1 / 8 x 1 / 4 x 1 / 2 + + + level 1 level 2 + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
  • 82. The Earth Mover’s Distance Approximations
  • 83. The Earth Mover’s Distance Approximations Shirdhonkar and Jacobs 08 - approximated EMD using the sum of absolute values of the weighted wavelet coefficients of the difference histogram. D i f f e r e n c e P Q Wavelet Transform j=0 j=1 Absolute values + £ 2 ¡ 2 £ 0 £ 2 ¡ 2 £ 1
  • 84. The Earth Mover’s Distance Approximations Khot and Naor 06 – any embedding of the EMD over the d-dimensional Hamming cube into must incur a distortion of . Andoni, Indyk and Krauthgamer 08 - for sets with cardinalities upper bounded by a parameter the distortion reduces to . Naor and Schechtman 07 - any embedding of the EMD over must incur a distortion of . L 1 ­ ( d ) s O ( l o g s l o g d ) f 0 ; 1 ; : : : ; ¢ g 2 ­ ( p l o g ¢ )
  • 85. Robust Distances Very high distances outliers same difference.
  • 86. Robust Distances With colors, the natural choice. 0 0 20 40 60 80 100 120 CIEDE200 Distance from Blue
  • 87. Robust Distances ( b l u e ; r e d ) = 5 6 ¢E 0 0 ( b l u e ; y e l l o w ) = 1 0 2 ¢E 0 0
  • 88. Robust Distances - Exponent Usually a negative exponent is used: ² L e t d ( a ; b ) b e a d i s t a n c e m e a s u r e b e t w e e n t w o f e a t u r e s - a ; b . ² T h e n e g a t i v e e x p o n e n t d i s t a n c e i s : d e ( a ; b ) = 1 ¡ e ¡ d ( a ; b ) ¾
  • 89. Robust Distances - Exponent Exponent is used because (Ruzon and Tomasi 01) : robust, smooth, monotonic, and a metric Input is always discrete anyway …
  • 90. Robust Distances - Thresholded ² L e t d ( a ; b ) b e a d i s t a n c e m e a s u r e b e t w e e n t w o f e a t u r e s - a ; b . ² T h e t h r e s h o l d e d d i s t a n c e w i t h a t h r e s h o l d o f t > 0 i s : d t ( a ; b ) = m i n ( d ( a ; b ) ; t ) .
  • 91. Thresholded Distances Thresholded metrics are also metrics (Pele and Werman ICCV 2009). Better results. Pele and Werman ICCV 2009 algorithm computes EMD with thresholded ground distances much faster. Thresholded distance corresponds to sparse similarities matrix -> faster QC / QF computation. A i j = 1 ¡ D i j m a x i j ( D i j )
  • 92. Thresholded Distances Thresholded vs. exponent : Fast computation of cross-bin distances with a thresholded ground distance. Exponent changes small distances – can be a problem (e.g. color differences).
  • 93. Thresholded Distances Color distance should be thresholded (robust). 0 50 100 150 distance from blue ¢ E 0 0 m i n ( ¢ E 0 0 ; 1 0 ) 1 0 ( 1 ¡ e ¡ ¢ E 0 0 4 ) 1 0 ( 1 ¡ e ¡ ¢ E 0 0 5 )
  • 94. Thresholded Distances Exponent changes small distances ¢ E 0 0 m i n ( ¢ E 0 0 ; 1 0 ) 1 0 ( 1 ¡ e ¡ ¢ E 0 0 4 ) 1 0 ( 1 ¡ e ¡ ¢ E 0 0 5 ) 0 5 10 distance from blue
  • 95. A Ground Distance for SIFT d R = j j ( x i ; y i ) ¡ ( x j ; y j ) j j 2 + m i n ( j o i ¡ o j j ; M ¡ j o i ¡ o j j ) d T = m i n ( d R ; T ) T h e g r o u n d d i s t a n c e b e t w e e n t w o S I F T b i n s ( x i ; y i ; o i ) a n d ( x j ; y j ; o j ) :
  • 96. A Ground Distance for Color Image T h e g r o u n d d i s t a n c e s b e t w e e n t w o L A B i m a g e b i n s ( x i ; y i ; L i ; a i ; b i ) a n d ( x j ; y j ; L j ; a j ; b j ) w e u s e a r e : d c T = m i n ( ( j j ( x i ; y i ) ¡ ( x j ; y j ) j j 2 ) + ¢ 0 0 ( ( L i ; a i ; b i ) ; ( L j ; a j ; b j ) ) ; T )
  • 98. Perceptual Color Differences Euclidean distance on L*a*b* space is widely considered as perceptual uniform. 0 100 200 300 distance from blue j j L a b j j 2
  • 99. Perceptual Color Differences 0 20 40 60 adist2 distance from blue Purples before blues j j L a b j j 2
  • 100. Perceptual Color Differences on L*a*b* space is better. Luo, Cui and Rigg 01. Sharma, Wu and Dalal 05. distance from blue 0 5 10 15 20 ¢ E 0 0 ¢ E 0 0
  • 101. Perceptual Color Differences on L*a*b* space is better. ¢ E 0 0 j j L a b j j 2 ¢ E 0 0
  • 102. Perceptual Color Differences 0 50 100 150 distance from blue on L*a*b* space is better. ¢ E 0 0 ¢ E 0 0
  • 103. Perceptual Color Differences on L*a*b* space is better. But still has major problems. Color distance should be thresholded (robust). ( b l u e ; r e d ) = 5 6 ¢E 0 0 ( b l u e ; y e l l o w ) = 1 0 2 ¢E 0 0 ¢ E 0 0
  • 104. Perceptual Color Differences Color distance should be saturated (robust). 0 50 100 150 distance from blue ¢ E 0 0 m i n ( ¢ E 0 0 ; 1 0 ) 1 0 ( 1 ¡ e ¡ ¢ E 0 0 4 ) 1 0 ( 1 ¡ e ¡ ¢ E 0 0 5 )
  • 105. Thresholded Distances Exponent changes small distances ¢ E 0 0 m i n ( ¢ E 0 0 ; 1 0 ) 1 0 ( 1 ¡ e ¡ ¢ E 0 0 4 ) 1 0 ( 1 ¡ e ¡ ¢ E 0 0 5 ) 0 5 10 distance from blue
  • 107. Perceptual Color Descriptors 11 basic color terms. Berlin and Kay 69. white red green yellow blue brown purple pink orange grey black
  • 108. Perceptual Color Descriptors 11 basic color terms. Berlin and Kay 69.
  • 109. Perceptual Color Descriptors 11 basic color terms. Berlin and Kay 69. Image copyright by Eric Rolph. Taken from: http://upload.wikimedia.org/wikipedia/commons/5/5c/Double-alaskan-rainbow.jpg
  • 110. Perceptual Color Descriptors 11 basic color terms. Berlin and Kay 69.
  • 111. Perceptual Color Descriptors How to give each pixel an “11-colors” description ?
  • 112. Perceptual Color Descriptors Learning Color Names from Real-World Images, J. van de Weijer, C. Schmid, J. Verbeek CVPR 2007.
  • 113. Perceptual Color Descriptors Learning Color Names from Real-World Images, J. van de Weijer, C. Schmid, J. Verbeek CVPR 2007.
  • 114. Perceptual Color Descriptors Learning Color Names from Real-World Images, J. van de Weijer, C. Schmid, J. Verbeek CVPR 2007. chip-based real-world
  • 115. Perceptual Color Descriptors Learning Color Names from Real-World Images, J. van de Weijer, C. Schmid, J. Verbeek CVPR 2007. For each color returns a probability distribution over the 11 basic colors. white red green yellow blue brown purple pink orange grey black
  • 116. Perceptual Color Descriptors Applying Color Names to Image Description, J. van de Weijer, C. Schmid ICIP 2007. Outperformed state of the art color descriptors.
  • 117. Perceptual Color Descriptors Using illumination invariants – black, gray and white are the same. “ Too much invariance” happens in other cases ( Local features and kernels for classification of texture and object categories: An in-depth study - Zhang, Marszalek, Lazebnik and Schmid. IJCV 2007, Learning the discriminative power-invariance trade-off - Varma and Ray. ICCV 2007 ). To conclude: Don’t solve imaginary problems.
  • 118. Perceptual Color Descriptors This method is still not perfect. 11 color vector for purple (255,0,255) is: In real world images there are no such over-saturated colors. 0 0 0 0 0 0 0 0 0 1 0
  • 119. Open Questions EMD variant that reduces the effect of large bins. Learning the ground distance for EMD. Learning the similarity matrix and normalization factor for QC.
  • 120. Hands-On Code Example http://www.cs.huji.ac.il/~ofirpele/FastEMD/code/ http://www.cs.huji.ac.il/~ofirpele/QC/code/
  • 121. Tutorial: Or “Ofir Pele” http://www.cs.huji.ac.il/~ofirpele/DFML_ECCV2010_tutorial/

Editor's Notes

  • #13: Endres, D. M.; J. E. Schindelin (2003). &amp;quot;A new metric for probability distributions&amp;quot;. IEEE Trans. Inf. Theory 49 : pp. 1858–1860. doi : 10.1109/TIT.2003.813506 . Ôsterreicher, F.; I. Vajda (2003). &amp;quot;A new class of metric divergences on probability spaces and its statistical applications&amp;quot;. Ann. Inst. Statist. Math. 55 : pp. 639–653. doi : 10.1007/BF02517812
  • #15: 3 – Rubner’s thesis + Pele and Werman ECCV 2010
  • #33: Pele and Werman ECCV 2010
  • #34: Pele and Werman ECCV 2010
  • #35: Pele and Werman ECCV 2010
  • #36: Pele and Werman ECCV 2010
  • #37: Pele and Werman ECCV 2010
  • #38: Pele and Werman ECCV 2010
  • #39: Pele and Werman ECCV 2010
  • #40: Pele and Werman ECCV 2010
  • #41: Pele and Werman ECCV 2010
  • #42: Pele and Werman ECCV 2010
  • #71: Full net
  • #72: Full net
  • #73: Full net
  • #74: Full net
  • #75: Full net
  • #76: Full net
  • #77: Full net
  • #79: T = #random placements of grids N = #elements in histogram
  • #88: Graph fonts changed to 24