SHORT LISTING LIKELY IMAGES USING PROPOSED MODIFIED-SIFT TOGETHER WITH CONVENTIONAL SIFT FOR QUICK SKETCH TO PHOTO MATCHING

International Journal in Foundations of Computer Science & Technology (IJFCST), Vol. 3, No.4, July 2013
DOI:10.5121/ijfcst.2013.3406 51
SHORT LISTING LIKELY IMAGES USING PROPOSED
MODIFIED-SIFT TOGETHER WITH CONVENTIONAL
SIFT FOR QUICK SKETCH TO PHOTO MATCHING
Mohd. Ahmed1
, S. A. Patekar2
, Mandar Sohani3
1, 2, 3
Vidyalankar Institute of Technology, wadala east, Mumbai - 400 037, India
1
smaengg@gmail.com,2
sunil.patekar@vit.edu.in,3
mandar.sohani@vit.edu.in
ABSTRACT
The paper proposes the modified-SIFT algorithm which will be a modified form of the scale invariant
feature transform. The modification consists of considering successive groups of 8 rows of pixel, along the
height of the image. These are used to construct 8 bin histograms for magnitude as well as orientation
individually. As a result the number of feature descriptors is significantly less (95%) than the standard
SIFT approach. Fewer feature descriptor leads to reduced accuracy. This reduction in accuracy is quite
drastic when searching for a single (RANK1) image match; however accuracy improves if a band of likely
(say tolerance of 10%) images is to be returned. The paper therefore proposes a two-stage-approach where
First Modified-SIFT is used to obtain a shortlisted band of likely images subsequently SIFT is applied
within this band to find a perfect match. It may appear that this process is tedious however it provides a
significant reduction in search time as compared to applying SIFT on the entire database. The minor
reduction in accuracy can be offset by the considerable time gained while searching a large database. The
modified-SIFT algorithm when used in conjunction with a face cropping algorithm can also be used to find
a match against disguised images.
KEYWORDS
magnitude histogram, orientation histogram, two-stage-approach, cropping algorithm, disguise image,
Modified-SIFT.
1. INTRODUCTION
Whenever a crime is committed by a criminal, he tries to run away from the spot as soon as
possible leaving behind trail of proofs, evidences and witnesses. With the help of witness’s
narration, forensic artist sketches the image of the criminal. Previously investigators used to
manually compare the sketches with mug shot database. It was very cumbersome and time
consuming method. Nowadays investigator applies various automated methods[1] [2] [3] [4] [5],
Algorithms having accuracy more than 90% are, FaceVACS, Sketch matching through SIFT,
matching through SIFT with LFDA [6] [7] [8]. FaceVACS is one of the best performing
commercial face recognition engine, however it is expensive and the accuracy is limited to 90%,
also it is specially design for face recognition and not sketch recognition[6][8].The most recent
technique is SIFT, A large number of descriptor is found through SIFT which is highly distinctive
[9]. This method gives us a high accuracy (97%) [8], however it takes very long search time [7].
Fast execution of task is of paramount importance and advantageous considering the huge
Database of criminals in populous countries. To improve the search time the paper proposes a
modification of SIFT. In Modified-SIFT features are reduced by taking 8 rows of pixel data

52
together for making 8 bin histogram. This 8 bin Histogram is a single Modified-SIFT feature. The
total number of feature in Modified-SIFT is much less. Thus we can see that in order to improve
search time accuracy has been reduced. This reduce accuracy is only for RANK1 photograph but
when tolerance of 10% images is to be returned then accuracy is good. Thus In order to resolve
the issues of accuracy and time spent this paper therefore proposes a two-stage-approach, wherein
a modification of SIFT (modified-SIFT) is used in conjunction with SIFT. In the first stage
Modified-SIFT is applied on large Database gallery to shortlist the database to ten percent
shortlisted photographs, then in the next stage SIFT is applied on these ten percent shortlisted
photographs. This two-stage-approach leads to a significant improvement in search time without
impairing accuracy. Considering a database of 3 lakh images the conventional SIFT approach
takes approximately 27hours [7], the two stage approach takes approximately about 3 hours 38
minutes. With ever increasing mugshot database sizes SIFT search time have been increasing
exponentially. It is in this domain that modified-SIFT holds great promise. The results of this
modified approach are also encouraging for changed appearance images, when used in
conjunction with face cropping algorithm [10].
2. The Proposed Modified-SIFT Algorithm
The proposed Modified-SIFT employs the following major stages of computation to generate
the feature descriptor of an image. As shown in fig.1.
Figure1: Modified-SIFT feature detection algorithm
2.1 Constructing a Scale Space
In the first step of Modified-SIFT, first of all, several octaves of the original images are
generated. Each octave’s image size is half the previous one. Within an octave, images are
progressively blurred using the Gaussian Blur operator. In the next step, all these octaves are used
to generate Difference of Gaussian images.
Mathematically, “blurring” is referred to as the convolution of the Gaussian Operator and the
image. Gaussian blur has a particular expression or “operator” that is applied to each pixel, which
results in the blurred image.
Take Difference of
Gaussians
Construct Scale Space
Filter Low Contrast
Responses
Locate DoG Extreama
Find orientation
histogram
Find magnitude
histogram
Go use your feature
Build keypoint
Descriptor

53
2.2 Finding Difference of Gaussian
Two consecutive images in an octave are picked and one is subtracted from the other.
Then the next consecutive pair is taken, and the process is repeated. This is done for all octaves.
The resulting images are an approximation of scale invariant Laplacian of Gaussian (which is
good for detecting keypoints). Next step is used to find some interesting keypoints, i.e. Maxima
of the image.
2.3 Locating Maxima in Difference of Gaussian Images
The first step is to closely locate the maxima. It is iterated through each pixel and scanning all
its neighboring pixels. The check is done within the current image, and also the one above and
below it. The maxima are detected in the difference of Gaussian images. Next, some insignificant
keypoints are rejected because they do not have enough contrast.
2.4 Generating Modified-SIFT Features
In the last step of the algorithm, the maxima image found previously used to calculate and find
the following:-
1) Histogram of magnitude image,
2) Histogram of orientation image,
Where magnitude and orientation images see fig.2 (b), (c), are found by applying the following
magnitude and orientation equation:-
… (1)
θ(x, y) = tan−1 ((L(x, y + 1) − L(x, y − 1)) / (L(x + 1, y) − L(x − 1, y))) … (2)
For successive eight rows following steps are followed to form the magnitude as well as
orientation histogram using magnitude and orientation images as in fig.2. (b) & (c) respectively.
2.4.1 Formation of Histogram of magnitude image
 First magnitude of every pixel of first eight rows of magnitude image is sorted.
 Next, minimum and maximum magnitude values are computed.
 These minimum and maximum values computed are the range of the bin of histogram.
 This range is divided into eight equal bins for creating a histogram.
 Finally, to create the histogram, all magnitude values within eight rows are categorized
into these eight bins, depending upon their values, they fall into compatible bin out of
eight bins.

54
(a) (b) (c)
Figure 2:(a) original photograph,( b) Magnitude image, (c) orientation image generating
descriptor on every 8 rows from top to bottom on (b) and (c), Red lines Shows 8 rows of magnitude and
orientation data of pixel.
2.4.2 Formation of Histogram of orientation image
 First orientation of every pixel of first eight rows of orientation image is sorted.
 Next, each pixel may obtain any value from zero (0) to 360 degree and NAN (Not a Null)
value.
 NAN is assigned as zero.
 And the range 0 to 360 degree is divided into eight equal bins as follows :-
0 – 44, 45 – 89, 90 – 134, 135 – 179, etc.
 Finally, to create the histogram, all orientation angle within eight rows are categorized
into these eight bins, depending upon their angle, they fall into compatible bin out of
eight bins.
3. Changes adopted over the conventional SIFT
 First two steps, namely finding scale space and difference of Gaussian, of Modified-SIFT
are similar to SIFT [9], the difference lies after the second step.
 In the third step of modified-SIFT only maxima is considered whereas in SIFT both
maxima and minima are considered as a keypoints. This results in a lower number of
keypoints being generated, which lowers the computational overhead.
 In the SIFT approach an additional overhead of removing keypoints along edges is
obtained, as it requires the use of an edge detection algorithm. In the modified-SIFT
approach keypoints along edges are not removed, saving this computational overhead.
However, it must be noted that descriptors formed in conventional SIFT after removal of
edge keypoints are much more distinct and gives SIFT its better accuracy.
 Next Modified-SIFT does not consider rotation invariance aspect of SIFT.
 Finally Creation of Modified-SIFT feature descriptor is totally different as compare to
SIFT. In an SIFT approach for calculating a feature descriptor, a 16 x 16 pixel window is
used for each of the calculated maxima. Each feature is a composite histogram consisting
of magnitude & orientation information. In the modified-SIFT approach successive
groups, of 8 rows of pixels, are considered, across the height of the image, to compute a
feature for each group. This consists of computing histograms for magnitude and
orientation separately. Also for SIFT, total number of feature generated is very large. For
each maxima, 16, 8 bin histogram i.e. 16 x 8 = 128 bins values are created, On an average

55
in a 200 x 250 image, approximately 400 maxima will be extracted in SIFT. This will
lead to 400 x 128 bins being created for comparison. Whereas in Modified-SIFT, for a
200 x 250 image (250 / 8 x 2) x 8 = 500, total bins will be created, Independent of the
number of maxima points.
In Short the time consume by SIFT in taking minima point and discretizing keypoints along edges
and thereby increasing computational overhead has been overcome by Modified-SIFT. Also in
Modified-SIFT, though the descriptor obtained is not vivid, lesser number of descriptors are
obtained, resulting in quickness in comparison.
4. The Proposed Two Stage Approach
In the first stage Modified-SIFT is used to reduce the search space, and then in the second stage,
SIFT is applied within this limited search space to find a match. For example, given a database of
3 lakh mugshot, Modified-SIFT would take 50 min to reduce the search space to approximately
30000 suspects (about 10%), subsequently SIFT would be used within the limited search space
and would find a match in 2 hour and 48 minutes. Thus the total time of search would be around 3
hour and 38 minutes.
4.1 Need for two stage Approach
In SIFT, feature are directly proportional to maxima points, i.e., more the number of maxima,
more is the number of features. Whereas in Modified-SIFT, features are independent of maxima
points. As such lesser number of features are detected, Since all the maxima points within 8 rows
are lumped together for simultaneous consideration, which shortens the search duration. Accuracy
is very good in band. But as the ratio of resemblance are vast in band, more images are
shortlisted, which is detrimental for the robustness and promptness of searching task.
It is at this point, SIFT is applied on the shortlisted images which are priorly obtained from
Modified-SIFT. Hence, upon fusion of Modified-SIFT and SIFT, the rate of accuracy has been
steadfastly maintained and at the same time, the search duration has been tremendously reduced.
Thus, it is evident that the good features of Modified-SIFT and SIFT are plucked and merged
together for the overall efficiency enhancement and useful application of the method.
4.2 Modified-SIFT to match disguised images
In order to match the disguised images following steps are followed:
 By using the face cropping Algorithm to crop the sketches as well as photographs in the
gallery.
 The images should be cropped from below, initiating from lips to forehead.
 Then Modified-SIFT is applied on the cropped images to find the feature descriptor.
 With the help of above method, feature descriptor of both sketches and photographs are
obtained.
 Then the feature descriptors of sketches are used to match vis-à-vis photograph in the
gallery.

56
5. Sketch Matching Results and comparison
5.1 Viewed Images
In order to find sketch matching through this proposed Modified-SIFT Algorithm, sketches and
photographs pair is taken from the CUHK data set [11]. Two examples of the viewed sketches are
shown in the Fig.3. (a) & (b).
(a) (b)
Figure 3. a), b) viewed sketch
For a given probe sketch and a particular database the following tables, 1,2 & 3 provides a
comparison of number of descriptor, scanning speed & accuracy respectively while using SIFT
& Modified-SIFT.
Table 1. Total number of feature comparison
Sr. No. Image Size Total pixel SIFT feature[9]
(Approximately)
Modified-SIFT
Feature
1 100 x 125 12500 100 x 128 (100/8x2)x8=200
2 200 x 250 50000 400 x 128 (200/8x2)x8=400
3 400 x 400 160000 1200 x 128 (400/8x2)x8=800
4 500 x 500 250000 2000 x 128 (500/8x2)x8=1600
Remark 1: Modified SIFT features are 95% less than that of SIFT.
Table 2. Scanning Speed comparison (Approximate calculation after Sr. No. 2)
Sr. No. Total images in
Database
Require time to match
SIFT [7] Modified-SIFT
1 88 30Sec 1Sec
2 300 1min 40sec 3Sec
3 3000 16min 40sec 30Sec
4 30000 2hr 48min 5min
5 3 lakh 27hr 50min
Remark 2: Reduce feature leads to quick retrieval of photograph

57
Table 3. Accuracy comparison
DB size SIFT[8]
Accuracy %
Modified-SIFT
Accuracy %
Modified-SIFT
Top 10% band
Accuracy
10 100 100 100
30 100 90 99
88 99 70 98
300 97 50 97
Remark 3: Modified-SIFT algorithm’s Accuracy in 10% band is approaches to that of SIFT
Table4 shows the Results of two-stage-approach for relative comparison of SIFT and
modified-SIFT regarding search speed and its accuracy.
Table 4. Two-stage-approach, search time (Approximate calculation after Sr. No. 2)
Sr.
No.
DB
size
Require time to match in two stage approach
Stage 1
Modified-SIFT applied to
obtain reduce DB(10% of
DB)
Stage 2
Conventional
SIFT on
reduce Search
Space
Total search time
Search
Time(T1)
Reduced
Search Space
Search
Time(T2)
Total Search Time
T=T1+T2
1 88 1sec 9 3sec 4sec
2 300 3sec 30 10sec 13 sec
3 3000 30sec 300 1min 40sec 2min 10 sec
4 30000 5min 3000 16min 40sec 21min 40sec
5 3 lakh 50min 30000 2hr 48min 3hr 38min
Remark 4: Two stage approach search time is nearly 7 times faster than the conventional SIFT.
5.2 Disguise Images
Modified-SIFT also shows promising results (Appendix A) while searching disguised images, a
domain in which SIFT algorithms falter. However, for this purpose face cropping Algorithm must
be used in conjunction with Modified-SIFT.
6. CONCLUSIONS
The proposed Modified-SIFT method is extremely quick in analyzing the Database. Upon close
observation it has been found that it is almost 30 times faster than the contemporary software
such as SIFT. The percentage of accuracy has been traded off in lieu of speed. To achieve the
speed during scrutinizing method the number of relevant feature has been reduced. Our proposed
Modified-SIFT can be used exclusively or in complement with other method such as SIFT using
two stage approach. If used in complementary form speed and accuracy can be delicately
optimized.
Attendant advantages of Modified-SIFT include detection in spite of disguised and
impersonating facial changes. However, for detection in such situations, facial cropping
algorithms have to be used in conjunction with Modified-SIFT. Thus complementing the

58
modified-SIFT with conventional SIFT holds a considerable promise in terms of reducing the
search time substantially without impairment in accuracy.
REFERENCES
[1]X. Tang and X. Wang, “Face Sketch Recognition,” IEEE Trans. Circuits and Systems for Video
Technology, vol. 14, no. 1, pp. 50-57, Jan. 2004.
[2]Q. Liu, X. Tang, H. Jin, H. Lu, and S. Ma, “A Nonlinear Approach for Face Sketch Synthesis and
Recognition,” Proc. IEEE Conf. Computer Vision and Pattern Recognition, pp. 1005-1010, 2005.
[3] J. Zhong, X. Gao, and C. Tian, “Face Sketch Synthesis Using E-HMM and Selective Ensemble,” Proc.
IEEE Conf. Acoustics, Speech, and Signal Processing, 2007.BP sythesis
[4]X. Wang and X. Tang, “Face Photo-Sketch Synthesis and Recognition,” IEEE Trans. Pattern Analysis
and Machine Intelligence, vol. 31, no. 11, pp. 1955-1967, Nov. 2009. Eigen Transform
[5]X. Tang and X. Wang, “Face Sketch Synthesis and Recognition,” Proc. IEEE Int’l Conf. Computer
Vision, pp. 687-694, 2003.
[6]FaceVACS Software Developer Kit, Cognitec Systems GmbH, http://www.cognitec-systems.de.
[7]B. Klare and A. Jain, “Sketch to Photo Matching: A Feature-Based Approach,” Proc. SPIE Conf.
Biometric Technology for Human Identification VII, 2010.
[8]Brendan F. Klare, Zhifeng Li, Anil K. Jain-“Matching Forensic Sketches to Mug Shot Photos”, IEEE
Trans. Pattern Analysis and Machine Intelligence, VOL. 33, NO. 3, MARCH 2011.
[9]D. Lowe, “Distinctive Image Features from Scale-Invariant Keypoints,” Int’l J. Computer Vision, vol.
60, no. 2, pp. 91-110, 2004.
[10]Mr. Mayur S. Burange, Prof. S.V.Dhopte “Neuro Fuzzy Model for Human Face Expression
Recognition” IOSR Journal of Computer Engineering ( IOSRJCE ) ISSN : 2278-0661Volume1 Issue2
(May - June 2012) PP 01-06
[11]The CUHK Face Sketch Database is available for download at:
http://mmlab.ie.cuhk.edu.hk/facesketch.html.
Appendix A
A.1. Disguised image Database
In this study data set consisting of 10 bearded sketches is considered for each with corresponding
photograph and vice versa. All these sketches/photographs are taken from CUHK data base and edited with
the help of www.pizap.com and www.mustachfied.com online photograph editing software, for changing
the appearances. Two examples of the bearded sketches are shown in the figure 4. a, b.
(a) (b)
Figure 4. a), b) changed appearance sketches/photograph; sketch without beard and gallery photo with
beard and vice versa

59
A.2. Result
When Modified-SIFT applied directly on disguised images result is very poor as can be seen from the
table 5.
Table 5 Accuracy without cropping algorithm
DB size
Disguise image accuracy
10 10%
This result has been improved by first applying face cropping algorithm then Modified-SIFT is applied
on cropped images, See table6
Table 6 Accuracy with cropping algorithm
DB size Disguise image accuracy
10 40%
Two different cases of disguised images are considered Table7 shows the results of both the cases.
a. Sketch with beard and gallery photograph without beard.
b. Gallery photographs with beard and sketch without beard.
Table 7 Accuracy of both the cases with cropping algorithm
DB size Case I accuracy Case II accuracy
10 40% 50%
Authors
Mohd. Ahmed is currently Assistant professor at the M.H. Saboo Siddik College of
Engineering and pursuing M.E. from Vidyalankar Institute of Technology (VIT),
Mumbai and formerly trainee hardware & network engineer in DSS Mobile
communication, Mumbai. He is a B. E. in Computer Engineering. He is having 8 years
teaching and 1 year industry experience.
Dr. S. A. Patekar is currently Principal at the Vidyalankar Institute of Technology
(VIT), Mumbai and formerly Professor and Head, Department of Computer Technology,
Veermata Jijabai Technological Institute (VJTI), Mumbai. He is a Chartered Engineer,
I.E.(I), and has a Ph.D. in electrical engineering. He is the recipient of the Lord Reay
Gold Medal and Sir Phirozeshah Silver Medal of VJTI. He was honoured with the Indian
Society for Technical Education National Award in 1992 and the Best Teacher Award by
the Government of Maharashtra in 2007.
Mandar Sohani is currently Associate professor at the Vidyalankar Institute of
Technology (VIT), Mumbai and formerly Electronic engineer in industry. He is a B. E.
in Electronics Engineering, and has M. Tech in Electronics and Telecommunication
Engineering. He is having 12 years teaching and 1 year industry experience. He has
strong acumen in diagnosis of microprocessor based system. His current field of work
lies in the domain of image processing.

SHORT LISTING LIKELY IMAGES USING PROPOSED MODIFIED-SIFT TOGETHER WITH CONVENTIONAL SIFT FOR QUICK SKETCH TO PHOTO MATCHING

More Related Content

Similar to SHORT LISTING LIKELY IMAGES USING PROPOSED MODIFIED-SIFT TOGETHER WITH CONVENTIONAL SIFT FOR QUICK SKETCH TO PHOTO MATCHING (20)

More from ijfcstjournal (20)

Recently uploaded (20)

SHORT LISTING LIKELY IMAGES USING PROPOSED MODIFIED-SIFT TOGETHER WITH CONVENTIONAL SIFT FOR QUICK SKETCH TO PHOTO MATCHING