Multilabel Image Retreval Using Hashing

Ms.Bhosale Surbhi
Multilabel Image Retrieval Using
Hashing

Contents
• Introduction and Motivation
• Literature survey
• Proposed Approach
• Methodology of Evaluation
• Performance Result Analysis
• Conclusions and Future Work
• References
Multilabel Image Retrieval Using Hashing 2

Introduction
• With the increasing amount of image data over the
web, it is necessary to find the images that belongs to
similar objects as in a query image, has gained large
interest.
• For example, a query Pluto in Google Images, the
results should contain two different types of images:
one is Pluto in the solar system; the other is just the
image of Pluto planet.

Multilabel Images
Multilabel Image Retrieval 4

Similarity-preserving hashing
• Similarity-preserving hashing is a popular nearest neighbor
search technique for image retrieval.
• A representative stream of similarity-preserving hashing is
learning-to-hash, i.e., learning to compress data points (e.g.,
images) into binary representations such that semantically
similar data points have nearby binary codes.
• existing learning hash methods can be divided into two main
categories:
– supervised methods
– unsupervised methods.

An Illustration
Figure. 1. Illustration of instance-aware image retrieval.

Literature Survey

Objective
1. To study and analyse existing image retrieval systems.
2. To design a system to retrieve multilabel images.
3. To implement the proposed system.
4. To compare the performance of the proposed system
with that of existing systems.

Motivation
• String comparisons is time consuming.
• The learning of mapping relation between images is
required.
• It becomes difficult to find/retrieve candidate image
from whole image dataset because it consumes more
time and space.

Problem Statement
To design and develop multi label image retrieval
system using hashing.

Scope, Limitation and Restrictions
• The input to the system is an image database.
• The system accepts images of types JPEG, PNG, BMP.
• The size of image should not be less than 128 × 128.

13
Image Database
Region
Proposal
Deep Convolution Sub
Network Module/Spatial
Pyramid Pooling
Label Probability
Calculation Module
Instance Aware
Representation
Semantic
Hashing
Category
Aware Hashing
Populate Hash Code
Training Phase
Region
Proposal
Deep Convolution Sub
Network
Module/Spatial
Pyramid Pooling
Label Probability
Calculation Module
Category Aware
Retrieval
Testing Phase
Input Image Iq
Weight Transfer
Figure 2 : System Architecture

Data Set
1. VOC 2012 Dataset :
• There are twenty object classes:
1. person
2. bird, cat, cow, dog, horse, sheep
3. aero plane, bicycle, boat, bus, car, motorbike, train
4. bottle, chair, dining table, potted plant, sofa,
TV/monitor

Performance metric
The performance of the proposed system is measured in terms of precision and recall. If
M number of multilabel images are relevant from total K retrieved Multilabel images
then precision is given as,
If the database contains total N multilabel images relevant to the query image then recall is
given as,

Mathematical Model

Conclusion
• Deep network based hashing method is proposed which will be
able to label objects in an image.
• It combines similarity preserving hashing and category aware
hashing.
• Spatial pyramid pooling helps to represent image with fixed
length hash irrespective image's original size.
• The proposed system, methods like selective search greatly
improve regions of interest identification and thereby
segmentation problem.

Future Enhancement
• Crawling based dynamic hashed annotations of the each image on
the internet can be a possible solution of the problem.
• One can compute the hash code for any image on the fly as the web
crawler comes across it and update its references to similar image.
• This solution would also solve the problem of duplicate images on
the internet as same images would eventually end up having same
references.

References
[1] Hays, J. Efros, A. A. (2007). Scene completion using millions of photographs. ACM Transactions on
Graphics (TOG).
[2] Stein, B., zu Eissen, S. M. Potthast, M. (2007, July). Strategies for retrieving plagiarized documents.
In Proceedings of the 30th annual international ACM SIGIR conference on Research and development in
information retrieval (pp. 825-826). ACM.
[3] Datar, M., Immorlica, N., Indyk, P. Mirrokni, V. S. (2004, June). Locality-sensitive hashing scheme
based on p-stable distributions. In Proceedings of the twentieth annual symposium on Computational
geometry ACM.
[4] Kulis, B. Grauman, K. (2009, September). Kernelized locality-sensitive hashing for scalable image
search. In Computer Vision, 2009 IEEE 12th International Conference on (pp. 2130-2137). IEEE.
[5] Krizhevsky, I. Sutskever, and G. E. Hinton. ImageNet classification with deep convolutional neural
networks. In Proc. NIPS, 2012.
[6] B. Kulis and T. Darrell. Learning to hash with binary reconstructive embeddings. In Proc. NIPS,
pages 1042â˘A ¸S1050,2009.

[7] Salakhutdinov, R. Hinton, G. (2009). Semantic hashing. International Journal of
Approximate Reasoning, 50(7), 969-978.
[8] Liu,W.,Wang, J., Kumar, S. Chang, S. F. (2011). Hashing with graphs. In Proceedings of the
28th International Conference on Machine Learning
[9] A. Torralba, R. Fergus, and Y.Weiss. Small codes and large image databases for recognition.
In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2008.
[10] K. Lin, H.-F. Yang, J.-H. Hsiao, and C.-S. Chen, Deep learning of binary hash codes for
fast image retrieval, in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. Workshops, Jun. 2015.
[11] V. E. Liong, J. Lu, G. Wang, P. Moulin, and J. Zhou. Deep hashing for compact binary
codes learning. In Proc. IEEE Conference on Computer Vision and Pattern Recognition
(CVPR), 2015.
[12] G. Hinton and R. Salakhutdinov. Reducing the dimensionality of data with neural
networks. Science, 313(5786):504507, 2006.
[13] H. Lai, Y. Pan, Y. Liu, and S. Yan. Simultaneous feature learning and hash coding with
deep neural networks. In Proc. IEEE Conference on Computer Vision and Pattern Recognition
(CVPR), 2015.
[14] J. R. R. Uijlings, K. E. A. van de Sande, T. Gevers, and A. W. M. Smeulders, Selective
search for object recognition, Int. J. Comput. Vis., vol. 104, no. 2, pp. 154171, Apr. 2013.

[15] F. Zhao, Y. Huang, L.Wang, and T. Tan, Deep semantic ranking based hashing for multi-label image
retrieval, in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., Jun. 2015, pp. 15561564.
[16] R. Xia, Y. Pan, H. Lai, C. Liu, and S. Yan. Supervised hashing for image retrieval via image
representation learning. In Proc. AAAI Conference on Artificial Intelligence (AAAI), pages 21562162,
2014.
[17] Felzenszwalb, P. F., Huttenlocher, D. P. (2004). Efficient graph-based image segmentation.
International Journal of Computer Vision, 59, 167-181.
[18] K. He, X. Zhang, S. Ren, and J. Sun, Spatial pyramid pooling in deep convolutional networks for
visual recognition, in Proc. Eur. Conf. Comput. Vis., 2014, pp. 346361.
[19] C. Szegedy et al. (2014). Going deeper with convolutions.
[20] Y. Wei et al. (2014). CNN: Single-label to multi-label
[21] Deep Learning by Adam Gibson by Josh Patterson (Chapter 4. Major Architectures of Deep
Networks)
[22] Hanjiang Lai, Pan Yan, Xiangbo Shu, YunchaoWei, and Shuicheng Yan Instance- Aware Hashing for
Multi-Label Image Retrieval VOL. 25, NO. 6, JUNE 2016

Thank You

Selective Search
• Selective Search method is used to extract object
proposals from an input image.
• The selective search use Hierarchical clustering Method
to group or to cluster the regions.
• Input for Hierarchical clustering Method is set of regions
that are generated using graph based image segmentation.
• Initial Region is formed using graph based image
segmentation.
• Hierarchical clustering covers the entire image without
having to visit each pixel or each possible region.

Figure 3 : Generate Region Proposal using Selective Search[14]
Back

Deep convolution Sub-network Module
• Google Net deep architecture is used for the object
categorization and object detection task for each of the
proposed region.
• To optimize the task of feature learning, instead of
repeating the procedure for each of the individual proposal
which becomes computationally impossible if number of
generated region proposals are very high, Spatial Pyramid
Pooling which generates a feature map of all proposals
simultaneously.
• Here, N automatically generated region proposals and map
each of the proposal into d-dimensional intermediate feature
vector.
• So N × d feature matrix as an output from this module.

Figure 4 : Structure of Deep Convolutional Neural Network [15]. An input image is
first transformed to a fixed size and then goes through five convolutional layers
and two pooling layer.

Figure 5 : Convolution and Pooling Operations[21]
back

Label Probability Calculation Module
• It learns probability of each region belonging to each class.
• If there are c class labels and generate a probability vector for each
proposal, then in Pi = (P1
i , · · · , Pc
i ) Pj
i signifies the probability of
proposed region i belonging to class j.
• First compress for each of proposed region i, its corresponding feature d-
dimensional intermediate feature vector which corresponds to ith row of
N x d feature matrix into c-dimensional vector, then consolidate all this
compressed individual vectors into one c-dimensional vector.
• For this use cross hypothesis max pooling function such that for each of
class its probability of being assigned to input image is maximum
probability of it being assigned to any of proposed region.
mj = max{M1
j , M2
j , · · · , MN
j }, ∀j = 1, · · · , c
• where mj is the consolidated probability corresponding to category j. It
use each of this mj for all class categories to calculate probability
distribution p = (p1 , p2, . . . ,pc) given as,

back
Figure 6 : Illustration of the proposed label probability calculation module.
For an image (in c classes) with N region proposals[20]

Hash Coding Module
• First it convert input image into an instance
aware representation using cross-proposal
fusion.
• Then it will be used to do either of the
category-aware hashing or semantic hashing.

Cross Proposal Fusion
• Initially N x d matrix which was output of label
probability calculation module is compressed into b
dimensional matrix using fully-connected layer of
neural net.
• N x c probability matrix and N x b matrix is fused into
a c x b matrix using Kronecker product such that each
row corresponds to b-dimensional feature
corresponding to each class category, call as c x b
matrix as f,
back

Category Aware Hash Representation
For each image I we will compute c triplet loss functions
corresponding to each class category. This triplet loss
function is given as,
• I+ is any image that belongs to the same category as image I.
• I- is any image that doesn't belongs to the same category as image
I.
• f is organized into c groups. Then each f will be converted into b-
bit binary code
back

Semantic Hash Representation
• First it the c x b matrix into q-dimensional S assuming
that maximum length of hash code is q-bits using
fully-connected neural net layer.
• Weighted Triplet loss function is used if triplet loss
function given as,
back

Figure 7 : Hash Code Representation

Category Aware Retrieval
• It create a hash table from images in database for retrieval.
• This hash table has c columns each corresponding to one of the
c classes. For each image in retrieval dataset, so it encode it into
b-bit binary hash code for each of c class categories.
• Now this b-bit hash code for images jth class category is added
to corresponding jth column in hash table if the respective value
in the matrix output of label probability module is greater than
certain threshold value
• Similarly during retrieval, using test query image, c pieces b-bit
codes will be generated and drop ones which have probability
values less than 0.5.
• Now the search will be conducted in corresponding columns of
hash table to obtain a list of retrieved images.
back

Multilabel Image Retreval Using Hashing

More Related Content

What's hot (19)

Similar to Multilabel Image Retreval Using Hashing (20)

Recently uploaded (20)

Multilabel Image Retreval Using Hashing

Editor's Notes