A Survey on Exemplar-Based Image Inpainting Techniques

IJSRD - International Journal for Scientific Research & Development| Vol. 2, Issue 08, 2014 | ISSN (online): 2321-0613
All rights reserved by www.ijsrd.com 150
A Survey on Exemplar-Based Image in painting Techniques
Seema Dixit1
Saranjeet Singh2
1
M.Tech. Scholar 2
Assistant professor
1,2
Galaxy Global Group of Institutions, Ambala
Abstract— Preceding paper include exemplar-based image
inpainting technique give idea how to inpaint destroyed
region such as Criminisi algorithm, patch shifting scheme,
search region prior method. Criminsi’s and Sarawut’s patch
shifting scheme needed more time to inpaint an damaged
region but proposed method decrease time complexity by
searching only in related region of missing portion of image.
Keywords: image inpainting, Exemplar-based, object
removal, patch
I. INTRODUCTION
The image inpainting technology is a hotspot in computer
graphics, nowadays. And it has importance in heritage
preservation, film and television special effects production,
removing extra objects etc. in the museums, this concept is
used for damaged painting. The filling of lost information is
essential in image processing, with transmission, special
effect and image reproduction. The basic idea of filling lost
information with available information from their
environment (Bertalmio, et at, 2003) [1]. Inpainting is
basically a time consuming process it was the manual
process. It has number of applications like red eye
correction, object removal from digital photographs, image
coding, super resolution and transmission. Image Inpainting
reconstruct damage part of an image by carried out
information from its surrounding.
Image inpainting may be called as modification and
manipulation of an image.fig.1 shows an example for this
technique, there is a tree which is the target region which is
replace by information from remaining part of the image in
a plausible way. The algorithm automatically does such that
it looks like reasonable to the human eye. Detailed that are
hidden completely by the object to be removed can’t be
recovered by any mathematical method. So, the objective of
the image inpainting is not to recover the original image, but
to create some image that has resemblance with the original
image (Anupam, et al).The main goal of this process is to
reconstruct damaged parts or missing parts of image. And
this process reconstructs image in such a way that the
inpainted region cannot be detected by a casual observer.
Inpainting technique has found widespread use in many
applications such as reconstruction of old films, object
removal in digital photos, red eye correction, super
resolution, compression, image coding and transmission (A.
Criminisi, et al, 2003)[2].Diffusion based Inpainting was the
first fundamental digital Inpainting approach. In this
approach, lost region is filled by diffusing the image
information from the known region into the missing region
at the pixel level. Basically algorithms are based on theory
of variation method and Partial Differential equation (PDE).
The diffusion based Inpainting algorithm produces superb
results or filling the non-textured or relatively smaller
missing region. Later Chan et al. (2001) gives total variation
(TV) inpainting method. [5] And curvature-driven diffusion
(CDD) model. [6] The disadvantage of the diffusion process
is that it introduces some blur, which are noticeable when
filling larger regions. All the PDE based inpainting models
are more reliable for completing small, non- textured target
region. The second category of Inpainting is exemplar-based
Inpainting algorithm. This method of image Inpainting is an
efficient approach to reconstructing large target region.
Exemplar-based Inpainting approaches iteratively
synthesize the target region by almost similar patch in the
source region. Cheng et al. (2007) gives improved method
that provided mere reasonable priority function. [7] And
Wexler (2007) introduced global optimization to this
technique [4].These algorithms also remove the drawbacks
of PDE based inpainting. It also removes smooth effect of
the diffusion based Inpainting algorithm.
Gunamani et al in (2010) has presented an
inpainting algorithm, which is used to fill the broken region
with significant results. Many algorithms that are
presented before generally required some minutes for
inpainting the slighter areas on present personal
computers, it a time consuming.[8] That time is not
satisfactory for interactive sessions and motivated us to
propose a simpler and faster algorithm able to producing
equivalent outcomes within a few seconds. The
experimental consequences created by the algorithm are two
to three in the orders of magnitude faster than the existing
one.
Fig. 1: Removed large objects from image. (a) Original
image. (b) The region corresponding to the foreground tree
has been manually selected and then automatically removed.
(A. Criminisi, et at, 2003).
II. EXEMPLAR- BASED APPROACH
Image inpainting is modeled to fill the missing region ,
called as target region, of the given image part by the
information of the known region, known as source
region / or . In exemplar- based image inpainting
technique, in order to fill the target patch Ψp, which is
centered at pixel point p and partially within area Ω, the best
match patch ̂, which is situated at̂, is chosen from the
source region. Then the intensities of Ψp, in the target

(IJSRD/Vol. 2/Issue 08/2014/037)
region are completed by copying from the corresponding
pixels of ̂. The order of choosing target patch intensively
affects the reconstructed result as the example shown in [2].
For a natural looking result, the edges should be
continued which means that the patch that contains high
structural information should be filled first. With this
principle, patch priority P is introduced [2].
It is determined by the magnitude of the isophote
direction and the known pixel in the target patch. The target
patch which has the highest patch priority is filled,
mathematically, patch priority is defined as
P (p) = C (p). D (p) (2.1)
The confident term C (p) and data term
D (p) is defined as,
C (P) =
∑
| |
and
D (p) =
| |
(2.2)
Where |Ψp| is the area of Ψp, np is the normal
vector of the front δΩ, is the isophote at p and α is the
normalizing factor which equals 255 for 8-bit grey-scale
image. The confident term C shows the ratio of known
pixels surrounding at the center of target patch. The data
term D shows the strength of the edge at the target patch.
The process of Exemplar-based approach can be detailed as
follows. Firstly, the confident term is initialized by
assigning to C (p) =0 for p Ω and C (p) =1 for p
Then the following processes are repeated until the filling
front δ t
=
(1) Identify the filling front δΩ.
(2) Compute patch priorities of all the patches those
centers align on filling front δΩ.
(3) 3. Chose the patch Ψp which has maximum patch
priority.
(4) Find the best match patch Ψ̂ of from the
source region UΩ.
(5) Copying the data from Ψ̂ to Ψp for p Ψp Ω.
(6) Update C (q) for p Ψp Ω.
Note that, the best match patch Ψ̂ in step 4 is the
patch that minimizes the Sum of Squared Differences (SSD)
between itself and Ψp in known region. SSD is defined as
follows:
D ( = ∑ | |2
(2.3)
A. Search Region Prior
Traditional method, finds best patch using sum of squared
difference of small number of pixels between target patch
and candidate patch within fixed size of search area. When
the known pixels of target patch do not have sufficient
information to differentiate proper color or texture of correct
patch, it is possible that wrong candidate patch is chosen
within search region. Here, we have two approaches for this
problem, to have a target patch to contain more meaningful
features representing adjacent known region, or to provide
better search region with consistent color and texture.
Sarawut’s et. al. (2011) patch shifting scheme [3] is related
to the first approach. This method shifts target patch toward
source region, so the target patch can have more number of
known pixels.
It is still possible that increasing known pixels in
target patch that does not contain enough information to
distinguish correct patch in search area.
Instead of adding more information in the target
patch, this method provide better search region excluding
any sub region with inconsistent color or texture compared
to known pixels of target patch. Following is the process:
Step1. Divide the given input image using color or
texture, based on input image, proper image partitioning
method can be chosen.
Step2. Generate region index map by setting
different index to all pixels in each region as shown in figure
(3).
Step3.Compute the patch priority for all patches
that center is located along the boundary of , and then
choose the patch with maximum priority value.
Step4. Next, Find the best match patch, , that has
maximum sum of squared difference (SSD) with known
pixel point p in the target patch, where ( ) As
in step 2.3
Except following candidate patch condition. For
searching best patch, search region is selected with the
following valid candidate patch criterion:
Ψc for q: R (p) = R (q), q , p ( ) (2.2.1)
Where Ψc is the valid candidate patch in search
region, R (p) is the region index at point p is known pixels
of target pitch (q) is region index at point q in candidate
patch
This condition implies that only candidate patch in
search area which have the same region indices with target
patch are considered

Fig. 3: selecting search area setting: (a) region index map
from search region partition. (b); (d); (f) search area of
traditional method. (c); (e); (g) search area of search region
prior method.
suppose that target patch is chosen from right
boundary of the house in figure 3(a), traditional method uses
search area with fixed size in any case as shown in figure
3(b),(d)and (f).this method select search region based on
region index involving in the target patch, Ψp, for example,
if known pixels of target patch contains region index 4 as
figure3(d),region of index 4 as in figure 3(c) is excluded
from search area.
When known pixels of target patch contain more
than two region indices as in figure 3(g),search area is set to
sum of only candidate patch that have all corresponding
indices. In this way, only candidate patch close to the edge
is searched in figure3 (g), excluding other patches far from
the edge.
Step5. Search the best match patch of ̂ to
from the source region UΩ, copy data from Ψ̂ to for
Ψp Ω and Update C (q) for Ω.
Step6. Update region index map by coping region
index values from best patch, Ψq, to unknown pixel point P
in target parch, where Ω.
B. Patch Shifting Scheme
The best result of Exemplar-based approach may not be
achieved some cases, the target patch has not enough known
pixels for a meaningful representation. This problem can
occur and destroy the final result. So the number of known
pixels is a parameter to consider the patch priority.
In figure.5 it shows that the target patches on right
side would produce much better result than the target
patches present in left side. Here, we present an easy but
efficient approach to modify the target patch in such a way
that it always contains enough known pixels to produce
more reliable result.
The basic idea of this scheme is to shift the target
patch to the known region in the case that there are not
enough known pixels in that patch.
The sum of squared distance between the matched
patch ̂ and Ψp in known region is defined as: in equation
2.1,2.2,2.3
Fig. 4: experimental result: (a) original image. (b) Missing
region. (c) Region index map. (d) Result of traditional
method. (e) Result of search region prior method.
As shown in first row of figure 5, if 15 numbers of
known pixels (60% of the patch size) are sufficient for the
criteria then the target patch is moved toward the right as
shown in right. In this way, we gain 5 more known pixels
(20% of the patch size). For better understanding, we again
consider the second row of figure 5. If known pixels are
more than 76% of the patch size in each target patch, then
target patch, then target patch should be shifted one pixel
right and one pixel down as in the bottom right of that
figure. After shifting, we gain 4 more pixels (16% of patch
size).
Fig. 5: the idea of patch shifting scheme.
For this, we can find the vertical shift Sv and
horizontal shift Sh of the patch as:
∑ ∑
,
∑ ∑
(2.3.1)
Where
=[ ],
= [ ]
(2.3.2)
Is a binary image whose pixel value is 0 at
known pixel and 1 at unknown pixel, and p= (i, j) is the
center of the target patch.
On Exemplar-based inpainting, patch shifting is
applied to the target patch with maximum priority whose
number of known pixels is less than the predetermined
threshold.
The target patch repeatedly shift the number of
known pixel is more than threshold. Hence, the best patch of
the shifted target patch is searched.

As, if the suitable target patch, which have known
pixels more than the predetermined threshold, can’t be
achieved while none of the pixels in shifted patch is in the
initial patch, the next target patch with less priority is
selected and do the patch shifting again if require.
These processes are repeated until satisfied target
patch is found. For maintain advantage of patch shifting
scheme, we can apply patch shifting to only limited number
of target patch, otherwise target patch with low priority
introduced discontinuity in the restored edge. When there is
no suitable patch present, then target patch with low priority
can be chosen.
Damaged image Result of Criminisi method
The Result of patch shifting scheme
Fig. 5: the comparison result of patch shifting scheme with
Criminisi method.
III. CONCLUSION
In this review paper we present a brief description of
Exemplar-based image inpainting l model techniques. That
are Exemplar-based approach, Criminisi’ algorithm search
region prior method, patch shifting scheme. Exemplar-based
approach is capable of propagating both linear structure and
two dimensional textures into filling missing region with a
simple, single algorithm. and the search region prior
method prevent the target patch with different color or
texture region and provide visually more natural result.
At last we discuss patch shifting scheme that
produced better visual appearance than Criminisi’
inpainting. In future, we will try to introduce more effective
and efficient methods for enhancing PSNR and reduce
complexity.
REFERENCES
[1] Bertalmio L.Vese, G.Sapiro, and S. Osher,
Simultaneous structure and texture image inpainting
IEEE Transactions on Image Processing, Vol.12,
No.8, pp.882-889, 2003.
[2] A.Criminisi, P.Perez, and K.Toyama. Object removal
and region filling by Exemplar-based inpainting.
IEEE Transactions on Image Processing, Vol.13,
No.9, pp.1200-1212, 2004.
[3] Sarawut’s Tae-o-sot, and Akinori Nishihara,
“Exemplar-based image inpainting with patch
shifting scheme,”2011 17th
International Conference
on Digital Signal Processing (DSP) pp.1-5.
[4] Y.Wexler, E.Shechtman, M.Irani, “Space -Time
Completion of Video, “IEEE Transactions and
Pattern Analysis and Machine Intelligence, Vol.29,
pp.463-476, 2007
[5] Q.Chen, Y.Zang and Y. Liu, “Image inpainting with
improved exemplar-based approach,” Multimedia
content Analysis and Mining, LNCS, 2007.
[6] T.F Chan, J.H. Shen Mathematical model for local
non-texture Inpainting [J]. SIAM Journal of applied
Mathematics, 2001.
[7] T.F. Chan, J.H Shen. Non- texture Inpainting by
curvature-driven diffusion (CDD). Journal of Visual
Communication and image representation, 2007
[8] Jena G., “Restoration of Still Images Using
Inpainting Techniques”, International Journal of
Computer Science and Communication Engineering,
vol.1, No 2, pp.71-74, July- Dec 2010
[9] I.A. Ismail, E.A. Rakh, S.I. Zaki, M.A. Ashabrawy,
M.K. Shaat, "Crack detection and filling, using
steepest descent method", International Journal of
Computer and Electrical Engineering, Vol. 1, No. 4,
October, 2009.
[10]Z. Xu and S. Jian, “Image inpainting by patch
propagation using patch sparsity,” IEEE Transactions
on Image Processing, Vol. 19, Pp. 1153-1165, 2010
[11]Gunamani Jena, "Restoration of Still Images
using Inpainting techniques”, International Journal
of Computer Science & Communication, Vol. 1,
No. 2, , Pp. 71-74, July- December 2010.

A Survey on Exemplar-Based Image Inpainting Techniques

More Related Content

What's hot (20)

Similar to A Survey on Exemplar-Based Image Inpainting Techniques (20)

More from ijsrd.com (20)

Recently uploaded (20)

A Survey on Exemplar-Based Image Inpainting Techniques