Video object tracking with classification and recognition of objects

Video Object Tracking with Classification and Recognition of Objects By Manish Khare Under the Supervision of Dr. Rajneesh Kumar Srivastava Department of Electronics and Communication, University of Allahabad First progress Presentation

Introduction about Object Tracking Object tracking is an important task within the field of computer vision[1]. There are three key steps in video analysis: Detection of moving objects of interest Tracking of such objects from frame to frame Analysis of object tracked to recognize their behavior The use of object tracking is pertinent in the tasks of: motion-based recognition, automated surveillance, video indexing, human-computer interaction etc. First Progress Presentation on Video Object Tracking with Classification and Recognition of Objects

Introduction about Object Tracking (contd..) Usual input for a motion analysis system is a temporal image sequence with a corresponding amount of processed data. Motion analysis requires comprehensive information about moving object(s) in video sequences, which include segmentation and tracking of individual object from occluded scene of video. Video Object Tracking combines two phases of processing: Recognition and Classification of moving objects. Tracking of moving objects. During the phase of recognition and classification of moving objects, we classify the type of object. (For e.g. object is Car, Human Body, some machine, etc…). First Progress Presentation on Video Object Tracking with Classification and Recognition of Objects

Application of Tracking System The areas where this recognition and tracking system surveillance can be used are: monitoring of people in crowded area such as Shopping Mall, Temples or several commercial buildings. monitoring of the people to ensure that they are within the norms such as in secured banking. military and police. educational and manufacturing industries. First Progress Presentation on Video Object Tracking with Classification and Recognition of Objects

Shortcoming in Existing Tracking Systems The existing methods often require special markers attached with the objects, which prevent the different applications, other problems with the existing tracking systems are- They depend on data from limited field of view. i.e. fixed camera with limited view. Human operators are required to monitor activities. Require a lot of human intervention to track the object the same object in case the multiple cameras are used. First Progress Presentation on Video Object Tracking with Classification and Recognition of Objects

Modules in My Work The goal of the my research is to track multiple objects from video sequence, The main parts of the proposed research are as follows- Automatic Segmentation in video sequence. Recognition of objects. Classification of objects. Detection and Removal of Shadows of objects in video. Tracking of objects in video. First Progress Presentation on Video Object Tracking with Classification and Recognition of Objects

Work Going on this time for my Work Presently I am working on following two modules of my proposed research work: Automatic Segmentation in video sequence. Detection and Removal of Shadows of objects in video. First Progress Presentation on Video Object Tracking with Classification and Recognition of Objects

Automatic Segmentation The goal of the first step of the proposed work is to segment image sequence automatically into regions that are meaningful with respect to our application. Presently I am trying to segment medical images (specially Medical images with some noise and blur), after this I will segment real images and then proceed to image sequences. First Progress Presentation on Video Object Tracking with Classification and Recognition of Objects

Image Segmentation Segmentation subdivides an image into its constituent regions or objects [2]. The level to which the subdivision is carried depends on the problem being solved. Image segmentation algorithms generally based on one of two basic properties of intensity values: Discontinuity Similarity First Progress Presentation on Video Object Tracking with Classification and Recognition of Objects

Image Segmentation (Contd..) Our hypothesis is to use multi-resolution automatic segmentation in combination with conventional edges and region based segmentations as well as Partial Differential Equation with level set methods. Level set methods have their own importance in segmentation due to its accuracy and fast speed [3]. I am using level set methods for my proposed work. First Progress Presentation on Video Object Tracking with Classification and Recognition of Objects

Level set method Introduced by Osher- Sethian in year 1988. Level set method is a way to denote active contours. Basically it is used in fluid mechanics, but it can also be used in imaging sciences. Vision and image segmentation Malladi-Sethian-Vermuri (1994) Chan and Vese (1999) Very few work have been done in image and video processing by using this technique so this is very challenging area. This depend on position, time, the geometry of interface, and the some energy function[4]. First Progress Presentation on Video Object Tracking with Classification and Recognition of Objects

Level set method (contd..) First Progress Presentation on Video Object Tracking with Classification and Recognition of Objects

Illustration of level set method and the contour change [Chieh-Ling Huang(2009)] First Progress Presentation on Video Object Tracking with Classification and Recognition of Objects

Level set method (contd..) Advantages: It can easily represent complicated contour changes, for ex. When the contour splits into two or develops holes inside. Easily know whether a point is inside or outside the contour by checking Ф value. How to evaluate the contour value: [Chieh-Ling Huang(2009] First Progress Presentation on Video Object Tracking with Classification and Recognition of Objects

Difference of Level set method from Fast Marching Technique Fast Marching technique are designed for problems in which energy function never changes sign during processing, so that the contour is always moving forward or backward[5]. Level set Methods are designed for problems in which energy function can be positive in some place and negative in others, so that the front can move forwards in some places in others[5]. Level set significantly slower than fast marching method. First Progress Presentation on Video Object Tracking with Classification and Recognition of Objects

Level set method based image segmentation (Literature Survey) Very few work have been done for image segmentation using level set method up to this time. Work done for image segmentation by using level set method. Active contour without edges [T.F.Chan et al.(2001)] [8] Level set evolution without re-initialization [Chunming Li et al.(2005)] [10] Moment based Level set method for image segmentation [Juan Jhou et al.(2006)] [14] Level set method for image segmentation based on Bayesian Analysis [Xu Jing et al. (2008)] [11] Edge extraction with level set method for image segmentation [Zhang Junru et al.(2008)] [13] Shape based Level set method for image segmentation [Chieh Ling Huang (2009)] [15] Level set method based on overall Information of image [Li Min et al.(2010)] [12] Distance Regularized Level set Evolution [Chunming Li et al.(2011)] [9] First Progress Presentation on Video Object Tracking with Classification and Recognition of Objects

Active Contour Without Edges This paper propose a model for active contour to detect objects in a given image based on technique of contour evolution. This model can detect objects whose boundaries are not necessarily defined by gradient Here, the initial curve can be anywhere in the images, the interior contour automatically detected. This is oldest method for images segmentation using level set method. First Progress Presentation on Video Object Tracking with Classification and Recognition of Objects

Shape based Level set method level set method proposed by chan and vese can not work well on some specific shape. So, this paper add the shape knowledge into segmentation method. This algorithm can work with medical images, temperature images etc. Algorithm for this method is: Establish the initial shape model. Initialize the level set method. Adjust the shape model by current level set function. Determine the contour of shape. Compute the distance of the contour of shape by dilation operation. Update the level set function. Repeat the step 3-6 until the object is not segmented. First Progress Presentation on Video Object Tracking with Classification and Recognition of Objects

Edge extraction with level set method This paper deals with remote sensing images using level set methods. There are several advantages and disadvantage of extract edges using this method- This method is very flexible, we can just extract the edge that we want and we can have some control, when the curves moves. When we use edge detectors, sometimes we will meet a lot of properties of how to track the edges, while using level set method, the curve is continuous which make it easy to be tracking. One disadvantage – because the algorithm is much more complicated, it take very long time for processing. First Progress Presentation on Video Object Tracking with Classification and Recognition of Objects

Level set evolution without Re-initialization This paper proposed a new Variational formulation for geometric active contours. This method can easily implemented by using simple finite difference scheme. Here larger time step can be used to speed up the curve evolution. Variational formulation consists of an internal energy term that penalizes the deviation of the level set function a signed distance function and a external energy term that derives the motion of the zero level set toward the desired image features, such as object boundaries. First Progress Presentation on Video Object Tracking with Classification and Recognition of Objects

Level set method based on Bayesian Classifier this paper deals with color images, in which level set method applied. Traditional level set method used gray level images. Traditional method only set gray-level gradient information as the stopping power of contour lines to defined velocity function, which will distort when applied into color images. This paper re-designed a speed function based on color gradient function to color image to replace the traditional method of gray scale gradient by combining regional color characteristics. By using Bayesian classification, we can improve the speed function as integrate the regional color characteristics make use of processing image, regional color information to access & determine the contour lines for reach at goal. First Progress Presentation on Video Object Tracking with Classification and Recognition of Objects

Results of some Existing Algorithms (Using Level Set Method) Initial Contour 200 Iteration 450 Iteration 650 Iteration Result of [Chunming et al. (2005)] Initial 100 400 600 Contour Iteration Iteration Iteration Results of [Li jun Zhang et al. (2006)] First Progress Presentation on Video Object Tracking with Classification and Recognition of Objects

Results of some Existing Algorithms (Using Level Set Method) Initial Contour 100 Iteration 200 Iteration 300 Iteration Result of [Chunming et al. (2009)] Results of [Xu Jing et al. (2008)] First Progress Presentation on Video Object Tracking with Classification and Recognition of Objects

Results of some other Existing Algorithms Bragman Algorithm for Segmentation of image First Progress Presentation on Video Object Tracking with Classification and Recognition of Objects

Results of some other Existing Algorithms (Contd.) Fast Global Minimization of the Active Contour model base Image Segmentation First Progress Presentation on Video Object Tracking with Classification and Recognition of Objects

Results of some other Existing Algorithms (Contd.) Image segmentation by boundary extraction method First Progress Presentation on Video Object Tracking with Classification and Recognition of Objects

Results of some other Existing Algorithms (Contd.) Global Region-Based Image Segmentation Method First Progress Presentation on Video Object Tracking with Classification and Recognition of Objects

My Proposed Approach for Image Segmentation Take RGB/GRAY Image as input image. Convert it into GRAY Image. Apply Gaussian Convolution function for smoothness. Apply Dirac function into smooth Image. Initialize Level set energy function in image region R. [Negative contour value inside the region, Positive contour value outside the region and Zero contour value is at the region.] For iteration 1 to user define Call level set function with updated value (achieved by previous iteration). Update contour with updated value. Stop for loop either when iteration completed or when all contour value reach at zero value(neither at positive contour value nor at negative contour value). Stop Process. First Progress Presentation on Video Object Tracking with Classification and Recognition of Objects

Achieved Result by Proposed Approach Original Image Initial Contour 15000 Iteration 22000 Iteration Original Image Initial Contour 14000 Iteration 20000 Iteration First Progress Presentation on Video Object Tracking with Classification and Recognition of Objects

Achieved Result by Proposed Approach (contd..) Original Image Initial Contour 12000 Iteration 15000 Iteration Original Image Initial Contour 12000 Iteration 15000 Iteration First Progress Presentation on Video Object Tracking with Classification and Recognition of Objects

Shadow Detection and Removal Shadow detection is the next important component of the proposed work. By detecting the shadow, we can reduce the possibility of partial occlusion problem easily. Shadow detection is an important step in video surveillance and monitoring system, shadow point are often classified as object points causing errors in segmentation and tracking of moving objects. For this approach, firstly I am trying to detect and remove shadow from still image, then I will proceed to images sequences (video). First Progress Presentation on Video Object Tracking with Classification and Recognition of Objects

Shadow Detection What is Shadow? A shadow is an area where direct light from a light source cannot reach due to obstruction by an object. A shadow occurs when an object partially or totally occludes direct light from a source of illumination. Types of Shadow. Self Shadow: This shadow occurs in the portion of a object which is not illuminated by direct light. Cast Shadow: This shadow is the area projected by the object in the direction of direct light. First Progress Presentation on Video Object Tracking with Classification and Recognition of Objects

A B C [Figure From Li Xu et. al. (2006)] A - shows scene image with both cast and self shadows; B - gives an example of cast shadow of two photographers on a grass field; C - shows an example of self shadow First Progress Presentation on Video Object Tracking with Classification and Recognition of Objects

Shadow Detection (Contd..) Self Shadow is again divided into two parts: Shading. Interreflection. Cast Shadow is again divided into two parts: Umbra. Penumbra. Umbra, is the darkest part of the shadow. In umbra, the light source is completely occluded. Penumbra, is the region in which only a portion of the light source is obscured by the occluding body. [Figure From Sanjeev Kumar et. al. (2010)] First Progress Presentation on Video Object Tracking with Classification and Recognition of Objects

Why the need for shadow removal? Shadows cause tracking, segmentation or recognition algorithms to fail. Shadows have proven to be a large source of error in the detection and classification of vehicles. Real images with shadows can‘t be used for image synthesis. First Progress Presentation on Video Object Tracking with Classification and Recognition of Objects

Literature Survey about Shadow Detection and Removal Very few work have done for shadow detection and removal in case of images up to this time. There are five approaches based on shadow properties to detect and removal of shadow: Model Based Technique. Image Based Technique. Colour/Spectrum Based Technique. Texture Based Technique. Geometry Based Technique. First Progress Presentation on Video Object Tracking with Classification and Recognition of Objects

Model Based Technique In Model based technique, the geometry and illumination of the scene are assumed to be known. This includes the camera localization, the light source direction, and the geometry of observed objects, from which a priori knowledge of shadow areas is derived. In most applications the geometry of scene and/or light sources are unknown [23]. This technique is oldest in all techniques. First Progress Presentation on Video Object Tracking with Classification and Recognition of Objects

Image Based Technique Image based technique makes use of certain image shadow properties such as colour (or intensity), shadow structure (umbra and penumbra), boundaries, etc., without any assumption about the scene structure. If any of that information is available, it can be used to improve the detection process performance. According to Salvador et. al. [24] shadow do not change the surface texture, surface marking tend to continue across a shadow boundary under general viewing conditions. According to C. Jiang et. al. [25], in some colour components (or combination of them) no change is observed whether the region is shadowed or not, this is invariant to shadows First Progress Presentation on Video Object Tracking with Classification and Recognition of Objects

Colour/Spectrum Based Technique The Colour based technique attempts to describe the colour change of shaded pixel and find the colour feature that is illumination invariant. K. Siala [26] consider the pixel’s intensity change equally in RGB colour components and a diagonal model proposed to describe the colour distortion of shadow in RGB space. Cucchiara [27] investigated the Hue-Saturation-Value (HSV) colour property of cast shadows, and it is found that shadows change the hue component slightly and decrease the saturation component significantly. The shadow pixels cluster in a small region that has distinct distribution compared with foreground pixels. The shadows are then discriminated from foreground objects by using empirical thresholds on HSV color space. First Progress Presentation on Video Object Tracking with Classification and Recognition of Objects

Texture Based Technique The principle behind the textural model is that the texture of foreground objects is different from that of the background, while the texture of the shaded area remains the same as that of the background. The several techniques have been developed by using this technique. The technique of Leone et. al. [17,18] is a good approach and it is based on the observation that shadow regions present same textural characteristics in each frame of the gray-level video sequence and in the corresponding adaptive background model. The technique proposed D. Xu. [28] include the generation of initial change detection masks and canny edge maps. First Progress Presentation on Video Object Tracking with Classification and Recognition of Objects

Geometry Based Technique Geometric Model based technique makes use of the camera location, the ground surface, and the object geometry, etc., to detect the moving cast shadow, The Hsieh [29], Gaussian shadow model was proposed to detect the shadow of pedestrian. The model is parameterized with several features including orientation, mean intensity, and center position of a shadow region with the orientation and centroid position being estimated from the properties of object moments. This technique some has similarity with the model based technique. First Progress Presentation on Video Object Tracking with Classification and Recognition of Objects

Results of Some Existing Algorithms Results of [Sanjeev Kumar et. Al. (2010)] This approach is based on Colour Based Technique First Progress Presentation on Video Object Tracking with Classification and Recognition of Objects

Results of Some Existing Algorithms (Contd..) A - Original images with shadows. B – The reconstructed shadow images based on our method. C - The recovered shadow free images. These are Results of [Li Xu et. al. (2006)] This approach is based on Image Based Technique. A B C First Progress Presentation on Video Object Tracking with Classification and Recognition of Objects

Results of Some Existing Algorithms (Contd..) Result of [Li Xu et. al. (2008)] This approach is based on Model Based Technique Result of [Clement Fredembach et. al. (2006)] This approach is based on Texture Based Technique. First Progress Presentation on Video Object Tracking with Classification and Recognition of Objects

My Proposed Approach for Detecting and Removing Shadow Take RGB/GRAY image with shadow. Remove noise by using any non-linear filter. Calculate average Colour of image for determine effect of shadow in each of the dimension of Colour. [colours in shadow regions have larger value than the average, while colours in non-shadow regions have smaller value than the average values.] Construct some threshold to extract shadow regions. Use level set method for evaluate energy function for shadow are removal. Finally we got shadow free image. First Progress Presentation on Video Object Tracking with Classification and Recognition of Objects

Result of this proposed approach and other modules/methods, which I will do up to next progress presentation will be show in next presentation. First Progress Presentation on Video Object Tracking with Classification and Recognition of Objects

References Alper Yilmaz, Omar Javed, Mubarak Shah: “Object tracking: A survey” ACM Computing Surveys (CSUR) Volume 38 Issue 4, 2006. Rafael Gonzalez and R. E. Woods: “Digital Image Processing” Pearson Education, 2nd edition, 2002. Stanley Osher,Nikos Paragios: “Geometric Level Set Methods in Imaging, Vision, and Graphics” Springer publication, 1st edition, 2003. Amar Mitiche, Ismail Ben Ayed: “Variational and Level Set Methods in Image Segmentation” Springer publication, 1st edition, 2011. J.A. Sethian: “Level Set Methods and Fast Marching Methods - Evolving Interfaces in Computational Geometry, Fluid Mechanics, Computer Vision, and Materials Science” , Cambridge University Press, 1999. M. Kass, A. Witkin, and D. Terzopoulos, “Snakes - Active Contour Models'' International Journal of Computer Vision, 1(4): 321-331, 1987. T. F. Chan & L. A. Vese: “A Multiphase level set framework for image segmentation using the Mumford and Shah model” International Journal of Computer Vision 50(3), 271–293, 2002. T. F. Chan & L. A. Vese: “Active contours without edges” IEEE Transactions on Image Processing, 10(2), 266-277, 2001. C. Li, C. Xu, C. Gui, and M. D. Fox: "Distance Regularized Level Set Evolution and its Application to Image Segmentation", IEEE Trans. Image Processing, vol. 19 (12), 2010. First Progress Presentation on Video Object Tracking with Classification and Recognition of Objects

References (Contd..) C. Li, C. Xu, C. Gui, and M. D. Fox: "Level Set Evolution Without Re-initialization: A New Variational Formulation", CVPR 2005. Xu Jing, Wu Jian, Ye Feng, Cui Zhi-ming: “A Level Set Method for Color Image Segmentation Based on Bayesian Classifier” Proceeding of International Conference on Computer Science and Software Engineering, 886 - 890, 2008. Li Min, Xu Xiangmin, Qian Min, Wang Zhuocai: “A Fast Level Set Segmentation Method Based on the Overall Information of Image” Proceeding of 2nd International Symposium on Information Engineering and Electronic Commerce (IEEC), 1-4, 2010. Zhang Junru, Jiang Xuezhonga: “Research on Edge Extraction With Level Set Method” International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences. Vol. XXXVII. Part B4. Beijing 2008. Juan Zhou, Lixiu Yao, Jie Yang, Chunming Li: “Moment based level set method for image segmentation” Proceeding of 15th IEEE International Conference on Image Processing (ICIP), 1069 - 1072, 2008. Chieh-Ling Huang: “Shape-Based Level Set Method for Image Segmentation” Proceeding of Ninth International Conference on Hybrid Intelligent Systems, HIS '09. 243-246, 2009. Li-jun Zhang, Xiao-juan Wu, Zan Sheng: “A Fast Image Segmentation Approach based on Level Set Method”8th International Conference on Signal Processing, 2006. A. Leone, C. Distante, F. Buccolieri: “A texture-based approach for shadow detection” Proceeding of IEEE Conference on Advanced Video and Signal Based Surveillance, 371 - 376, 2005. First Progress Presentation on Video Object Tracking with Classification and Recognition of Objects

References (Contd..) A. Leone, C. Distante, N. Ancona, E. Stella, P. Siciliano: “Texture analysis for shadow removing in video-surveillance systems” Proceeding of IEEE International Conference on Systems, Man and Cybernetics, 6325 - 6330 vol.7, 2004. Li Xu, Feihu Qi, Renjie Jiang, Yunfeng Hao, Guorong Wu, Li Xu, Feihu Qi, Renjie Jiang, Yunfeng Hao, and Guorong Wu.: “Shadow detection and removal in real images: A survey” Technical report, Shanghai Jiao Tong University, 2006. Li Xu, Feihu Qi, Renjie Jiang: "Shadow Removal from a Single Image," Proceeding of Sixth International Conference on Intelligent Systems Design and Applications (ISDA'06) Volume 2, 1049-1054, 2006. Sanjeev Kumar and Anureet Kaur: “Shadow Detection And Removal in Colour Images Using Matlab” International Journal of Engineering Science and Technology. Vol. 2(9), 4482-4486, 2010. C. Fredembach, G. Finlayson: “Simple Shadow Removal” 18th International Conference on Pattern Recognition(ICPR), Hong Kong, 832 - 835, 2006. D. Roller, K. Daniilidis and H. H. Nagel: “Model-based object tracking in monocular image sequences of road traffic scenes” International Journal of Computer Vision, Volume 10, Number 3, 257-281, 1993. E. Salvador, A. Cavallaro, T. Ebrahimi: “Shadow identification and classification using invariant color models” Proceeding of IEEE International Conference on Acoustics, Speech, and Signal Processing, (ICASSP '01), 1545 - 1548 vol.3, 2001. C. X. Jiang, M. O. Ward: “Shadow Segmentation and Classification in a Constrained Environment” CVGIP: Image Understanding Volume 59, Issue 2, 213-225, 1994. First Progress Presentation on Video Object Tracking with Classification and Recognition of Objects

References (Contd..) K. Siala, M. Chakchouk, F. Chaieb, O. Besbes: “Moving shadow detection with support vector domain description in the color ratios space” Proceedings of the 17th International Conference on Pattern Recognition (ICPR 2004), 384 - 387 Vol.4, 2004. R. Cucchiara, C. Grana, M. Piccardi, A. Prati: “Detecting moving objects, ghosts, and shadows in video streams” IEEE Transactions on Pattern Analysis and Machine Intelligence(PAMI), Vol. 25(10), 1337 - 1342, 2003. Dong Xu, Xuelong Li, Zhengkai Liu and Yuan Yuan: “Cast shadow detection in video segmentation” Pattern Recognition Letters Vol. 26(1), 91-99, 2005. Jun-Wei Hsieh, Wen-Fong Hu,Chia-Jung Chang and Yung-Sheng Chen: “Shadow elimination for effective moving object detection by Gaussian shadow modeling” Image and Vision Computing, Vol. 21(6), 505-516, 2003. First Progress Presentation on Video Object Tracking with Classification and Recognition of Objects

First Progress Presentation on Video Object Tracking with Classification and Recognition of Objects

Video object tracking with classification and recognition of objects

More Related Content

What's hot (20)

Viewers also liked (20)

Similar to Video object tracking with classification and recognition of objects (20)

Recently uploaded (20)

Video object tracking with classification and recognition of objects