![www.projectsatbangalore.com 09591912372
Published in IET Computer Vision
Received on 7th November 2013
Revised on 17th September 2014
Accepted on 8th October 2014
doi: 10.1049/iet-cvi.2013.0298
ISSN 1751-9632
Particle filter framework for salient object
detection in videos
Karthik Muthuswamy, Deepu Rajan
Centre for Multimedia and Network Technology, School of Computer Engineering, Nanyang Technological University,
50 Nanyang Avenue, N4-02C-92 639798, Singapore
E-mail: kart0028@e.ntu.edu.sg
Abstract: Salient object detection in videos is challenging because of the competing motion in the background, resulting from
camera tracking an object of interest, or motion of objects in the foreground. The authors present a fast method to detect salient
video objects using particle filters, which are guided by spatio-temporal saliency maps and colour feature with the ability to
quickly recover from false detections. The proposed method for generating spatial and motion saliency maps is based on
comparing local features with dominant features present in the frame. A region is marked salient if there is a large difference
between local and dominant features. For spatial saliency, hue and saturation features are used, while for motion saliency,
optical flow vectors are used as features. Experimental results on standard datasets for video segmentation and for saliency
detection show superior performance over state-of-the-art methods.
1 Introduction
The human visual system (HVS) does a remarkable job at
separating foreground from background clutter even when
the objects in the scene have never been encountered
before. This capability is often attributed to its ability to
quickly locate ‘attentive’ regions in the scene and
subsequently, to understand the rest of the scene. In
computer vision, the phenomenon of identifying attentive
regions in image and videos is known as salient region
detection, where the region may or may not represent a
single object. The two main approaches to salient region
detection are top-down and bottom-up. The former
approach is an object or task-oriented process and
incorporates a fair amount of training, while the bottom-up
approach is driven by contrast between features leading to a
measure of conspicuity of a region.
The distinguishing aspect between image and video
saliency is the temporal motion information in the latter that
introduces the notion of motion saliency in addition to
spatial saliency of images. Bottom-up saliency for videos
helps in foreground region detection by assigning a saliency
measure to a pixel to be spatially and temporally salient.
The most commonly used technique to easily segment the
foreground from the background is by applying a threshold
to the saliency map [1]. However, if the saliency map is not
accurate, this might result in spurious regions being
identified as foreground.
In this paper, we propose a method for detecting the most
salient object in a video based on a computationally simple
approach for the generation of spatio-temporal saliency
maps. The most salient object is identified by localising a
set of pixels using particle filters whose weights are
computed from the saliency maps and colour features. The
location of the particles is influenced by the saliency map
as well as by the colour information in the frame. The
spatio-temporal saliency map allows the particle filter to
converge faster on the most salient foreground object by
refining the weights of the particles while the colour map
provides the robustness in matching the reference and target
distributions. The spatio-temporal saliency map is generated
at the pixel-level in the original resolution as opposed to
other methods [2–4] that operate on a reduced resolution or
on patches to reduce the computation time. In the latter
case, the accuracy of the saliency maps is compromised
when resized to the original resolution while our pixel-level
computation to generate saliency maps has been shown to
generate accurate saliency maps at an average processing
time of eight frames per second on a frame resolution of
352 × 288.
2 Related work
In the context of videos, motion saliency plays a very
important role in identifying the final saliency measure, in
addition to the spatial saliency from individual frames.
Here, we present the relevant work in motion saliency and
refer the readers to [5] for a survey on spatial salient region
detection. Stimuli-based saliency detection in videos was
first presented by Itti and Baldi [6] where they propose that
salient events are unexpected or surprising. In their model,
they define surprise as a statistical deviation from a recent
history of visual stimuli. Mahadevan and Vasconcelos [7]
have extended the centre-surround hypothesis of saliency
detection to videos, to measure the spatio-temporal saliency
www.ietdl.org
428
& The Institution of Engineering and Technology 2015
IET Comput. Vis., 2015, Vol. 9, Iss. 3, pp. 428–438
doi: 10.1049/iet-cvi.2013.0298](https://image.slidesharecdn.com/particlefilterframework-151124120312-lva1-app6892/85/Particle-filter-framework-for-salient-object-detection-in-videos-1-320.jpg)
Particle filter framework for salient object detection in videos
- 1. www.projectsatbangalore.com 09591912372 Published in IET Computer Vision Received on 7th November 2013 Revised on 17th September 2014 Accepted on 8th October 2014 doi: 10.1049/iet-cvi.2013.0298 ISSN 1751-9632 Particle filter framework for salient object detection in videos Karthik Muthuswamy, Deepu Rajan Centre for Multimedia and Network Technology, School of Computer Engineering, Nanyang Technological University, 50 Nanyang Avenue, N4-02C-92 639798, Singapore E-mail: [email protected] Abstract: Salient object detection in videos is challenging because of the competing motion in the background, resulting from camera tracking an object of interest, or motion of objects in the foreground. The authors present a fast method to detect salient video objects using particle filters, which are guided by spatio-temporal saliency maps and colour feature with the ability to quickly recover from false detections. The proposed method for generating spatial and motion saliency maps is based on comparing local features with dominant features present in the frame. A region is marked salient if there is a large difference between local and dominant features. For spatial saliency, hue and saturation features are used, while for motion saliency, optical flow vectors are used as features. Experimental results on standard datasets for video segmentation and for saliency detection show superior performance over state-of-the-art methods. 1 Introduction The human visual system (HVS) does a remarkable job at separating foreground from background clutter even when the objects in the scene have never been encountered before. This capability is often attributed to its ability to quickly locate ‘attentive’ regions in the scene and subsequently, to understand the rest of the scene. In computer vision, the phenomenon of identifying attentive regions in image and videos is known as salient region detection, where the region may or may not represent a single object. The two main approaches to salient region detection are top-down and bottom-up. The former approach is an object or task-oriented process and incorporates a fair amount of training, while the bottom-up approach is driven by contrast between features leading to a measure of conspicuity of a region. The distinguishing aspect between image and video saliency is the temporal motion information in the latter that introduces the notion of motion saliency in addition to spatial saliency of images. Bottom-up saliency for videos helps in foreground region detection by assigning a saliency measure to a pixel to be spatially and temporally salient. The most commonly used technique to easily segment the foreground from the background is by applying a threshold to the saliency map [1]. However, if the saliency map is not accurate, this might result in spurious regions being identified as foreground. In this paper, we propose a method for detecting the most salient object in a video based on a computationally simple approach for the generation of spatio-temporal saliency maps. The most salient object is identified by localising a set of pixels using particle filters whose weights are computed from the saliency maps and colour features. The location of the particles is influenced by the saliency map as well as by the colour information in the frame. The spatio-temporal saliency map allows the particle filter to converge faster on the most salient foreground object by refining the weights of the particles while the colour map provides the robustness in matching the reference and target distributions. The spatio-temporal saliency map is generated at the pixel-level in the original resolution as opposed to other methods [2–4] that operate on a reduced resolution or on patches to reduce the computation time. In the latter case, the accuracy of the saliency maps is compromised when resized to the original resolution while our pixel-level computation to generate saliency maps has been shown to generate accurate saliency maps at an average processing time of eight frames per second on a frame resolution of 352 × 288. 2 Related work In the context of videos, motion saliency plays a very important role in identifying the final saliency measure, in addition to the spatial saliency from individual frames. Here, we present the relevant work in motion saliency and refer the readers to [5] for a survey on spatial salient region detection. Stimuli-based saliency detection in videos was first presented by Itti and Baldi [6] where they propose that salient events are unexpected or surprising. In their model, they define surprise as a statistical deviation from a recent history of visual stimuli. Mahadevan and Vasconcelos [7] have extended the centre-surround hypothesis of saliency detection to videos, to measure the spatio-temporal saliency www.ietdl.org 428 & The Institution of Engineering and Technology 2015 IET Comput. Vis., 2015, Vol. 9, Iss. 3, pp. 428–438 doi: 10.1049/iet-cvi.2013.0298