Real time vehicle counting in complex scene for traffic flow estimation using multi-level convolutional neural network

International Journal of Advanced Technology and Engineering Exploration, Vol X(XX)
ISSN (Print): 2394-5443 ISSN (Online): 2394-7454

1

Real-Time Vehicle Counting in Complex Scene for Traffic Flow Estimation
Using Multi-Level Convolutional Neural Network
Zulaikha Kadim1*
, Khairunnisa Mohd. Johari2
, Den Fairol3
, Yuen Shang Li4
, Hock Woon Hon5
Advanced Informatics Lab, MIMOS Berhad, Technology Park, Malaysia1
Advanced Informatics Lab, MIMOS Berhad, Technology Park, Malaysia 2
Advanced Informatics Lab, MIMOS Berhad, Technology Park, Malaysia 3
©2016 ACCENTS
Abstract
Accurate traffic data collection is crucial to the relevant authorities in ensuring the planning, design, and management of the
road network can be done appropriately. Traditionally, traffic data collection was done manually by having human observers
at the site to count the vehicle as it passes the observation point. This approach is cost-effective; however, the accuracy can’t be
verified and may cause danger to the observers. Another common approach is utilizing sensors that need to be installed
underneath the road surface to collect traffic data. The accuracy of the data reading from the sensor is highly dependent on the
sensor installation, calibration, and reliability which usually deteriorated over time. For these reasons, vision-based approaches
have become more popular in traffic flow estimation tasks. Nevertheless, conventional image processing techniques which
utilize background subtraction-based approach may face problems in complex highway environment where the number of the
vehicle is high, a large gap in vehicle sizes of different classes and high occlusion rate. Thus, in this paper, a real-time vehicle
counting in a complex scene for traffic flow estimation using a multi-level convolutional neural network is proposed. By
exploiting the capabilities of deep-learning models in delineating and classifying objects in an image, it is shown that the system
can achieve average counting accuracy of 97.53% and a weighted average of counting with classification accuracy of 90.2%
validated on 585 minutes of highway videos collected from four different cameras; viewing at different vehicle's angles. The
system is also capable of running in real-time.
Keywords
Complex urban, convolutional neural network, traffic flow estimation, vehicle counting and classification
1.Introduction
Due to the rising traffic congestions, information about the traffic conditions can be used by traffic management
centres in several ways, such as to synchronize traffic lights, to assist drivers in the route’s selection, and to assist
governments in intercity connections and new roads planning. Traffic management can generate many benefits to the
road users, the municipals and central governments, and the environment. Drivers could benefit with less time spent
in travelling in urban and rural road, while governments can acquire those data for analysis and used those data to
improve urban road management. From the perspective on caring for the environment, this effort could reduce the
emission of pollutants. Conventional techniques for estimating traffic flow, such as deploying the inductive loops,
sonar or microwave detectors to obtain traffic flow information, have their disadvantages such as high installation
cost, traffic disruption during installation or maintenance, and the failure to detect slow or static vehicles [1]. The
recent sensor and communication technology advancement allow traffic management centres as well as municipals to
monitor closely the conditions of the urban and rural roads, through developing systems for monitoring the traffic
flow, and collecting data on the traffic flow characteristics [2].
In recent year, vision-based systems and image-based sensors have been heavily deployed to collect on-the-road data.
These traffic videos are expected to provide on-the-road traffic condition including monitoring abnormalities such as
accidents by taking advantages of the device installed on-site. In comparison, vision-based systems, is easy to install
and can be easily upgraded, as the system could be redesigned and its functionalities improved. These days, these
vision-based systems are deployed to count and classify vehicles, measure the speed of vehicles, and identify of traffic
incidents [3].

Zulaikha et al.
2

Recently, many vision-based techniques are being deployed in intelligent transportation systems to provide traffic
information which could derived important clue for many intelligent traffic monitoring systems, such as traffic index
and traffic density. This information could then be used to control, optimize and manage traffic [4]. In 2014, Andrews
Sobral [5] proposed a simple and effective pipeline to count vehicles, using, at first, a background subtraction
technique to detect moving vehicles, and then a blob tracker to track multiple vehicles to avoid duplicated counting
for vehicles. In 2016, Liu et al. [6] improved this simple pipeline to be able to work in the compressed domain for
videos obtained in highway. In 2016, Xia et al. [7] used the expectation-maximization (EM) algorithm to improve the
segmentation quality of moving vehicles. In [8], Vikram et. al. used pattern matching to identify traffic density for
efficient traffic light management.
These earlier researches, in the context of intelligent transportation, were deployed to detect, track and count vehicles
in videos. In earlier papers, vehicles are detected 1) using the model-based methods which uses prior knowledge [9,
10], 2) using the deformable templates for matching targets against known vehicle models in input videos [11], and
3) using simpler features such as corners and edges [12]. As for vehicle tracking, approaches such as mean-shift [13],
Kalman filtering [14], and particle filtering [15] were deployed.
The success of deploying CNN-based techniques to detect objects such as using Faster RCNN [16], SSD [17] and
YOLO [18], has provide many researcher reasons to deploy these methods to re-look in these methods could be
deployed not only to estimate traffic flow [19], but also to other ITS related applications such as road marker and
potholes detection [20]. The urban traffic monitoring systems uniqueness, where vehicles may appear to be 1) blurred
due to vehicle speed, and 2) small due to occluded vehicles, requires careful consideration when deploying CNN-
based techniques to estimate traffic. To address the above challenges, in this paper, we propose a multi-level
convolutional neural network (mCNN) framework to estimate traffic in complex urban settings, i.e. able to handle
various illumination conditions and complication mounting scenario. Herewith, the uniqueness of urban environment
in acquisition and processing due to the instrumentation set-up could showcase the proposed mCNN framework for
real-time deployment. This framework includes five major modules: 1) pre-processing; 2) object detection; 3)
tracking; 4) object classification, and 4) quantification. To handle irrelevant image details, the image is first cropped.
The first deep CNN network is then utilized to segment and pre-classified. Later, these pre-classified regions are
tracked along the cropped area. Then, the second-deep CNN network is deployed to refine the classification before a
quantification module is deployed to deliver the counting results. To test the proposed framework, the quantitative
results on realistic videos obtained from different acquisition set-up well demonstrate the performance of the proposed
method. The main contributions of this study can be summarized into the following:
• An end-to-end systematic solution for vehicle counting in complex urban setting using CNN-based techniques.
• A novel mCNN framework which could process acquired videos in real-time for vehicles from different views
such as frontal or rear.
• A robust mCNN framework which could estimate traffic, i.e. count and track vehicles, in real-time.
The rest of the paper is organized as follows. Section 2 introduces the detail of the proposed multi-level convolutional
neural network framework. In Section 3, implementation details are described with extensive experimental results on
the complex urban setting acquired. Finally, we conclude the paper with future work in Section 4.
2. Multi-Level Convolutional Neural Network (mCNN) Framework
This paper proposes a framework based on CNN, as depicted in Error! Reference source not found., to detect and
count vehicles in complex urban settings. It comprises of five major modules: 1) pre-processing; 2) object detection;
3) tracking; 4) object classification, and 4) quantification. The goal of the pre-processing module is to eliminate
irrelevant image details. The object detection module segments a set of interesting regions, and these regions will be
pre-classified. Later, these objects are tracked along consecutive video frames and classified into different classes
before the quantification module delivers a video-based counting for cars, vans, trucks, large trucks, buses, or
motorcycles.

International Journal of Advanced Technology and Engineering Exploration, Vol 6(35)
3

Figure 1 Multi-level CNN (mCNN) Framework
2.1Pre-processing
From the acquired video sequences, it is possible to observe that irrelevant details appearing in images due to the
instrumentation set-up. Thus, to discard the irrelevant details appearing in images, a cropping operation is applied (
Figure 1).

Figure 1. Sample cropping images for processing (gray area)
2.2Detection
In this step, a CNN model is applied for detection, i.e. to extract only relevant detected object. These objects consist
of four relevant type of vehicle (car, truck, bus, bike or person) with minimum confidence value of 0.6. Two different
CNN models are deployed in this detection step, to test the system accuracy and performance. Both models are selected
as they can provide superior detection speed as compared to other models such as R-CNN [21] and fast R-CNN [22]
with good detection accuracy.
The first model is the SSD Multibox [17]. SSD Multibox which uses a single deep neural network is selected as it
could achieve a good balance between the speed performance and accuracy performance, by discarding the fully
connected-layers in VGG-16 and replacing with a set of auxiliary convolutional layers, i.e. SSD uses those auxiliary
convolutional layers to extract features at multiple scales. These extracted feature maps will be input into a
convolutional kernel to predict the bounding box and classification probability. The score of the class probability and
the offsets (being the 4 coordinates of a bounding box) is computed. Those scores which exceed the threshold point
would be used as the final box to locate the object. Pre-trained SSD model is obtained from [23], which has been
trained using COCO dataset [24]. Input image size of the model is 300x300.
The second model is the YOLOv3 [25], which uses a single neural network for both classification and localising the
object using bounding boxes. The network has 24 convolutional layers plus 2 fully connected layers. Pretrained
convolutional layers weights (first 20 layers) are taken from darknet53 model which is pretrained on Imagenet
classification dataset [2, 3]. Then YOLO model is constructed by adding 4 convolutional layers and two fully
connected layers and further trained using COCO dataset. YOLO only predict one type of class in each grid. Hence,
it would be struggling in detecting a small object or very close object. Pre-trained YOLO model is obtained from [26],
which also has been trained using COCO dataset. Input image size of the model is 416x416.
2.3Tracker
In this step, a tracking algorithm is proposed to track the detected vehicles in successive frames. Tracker is required
to ensure that each vehicle is counted only once. The bounding box information of detected vehicles from vehicle
detection step will be used to initialize the tracker for the first time. In successive frames, detected vehicles information
will be used to update the trackers states. In the event of missing observation from detection step, the tracker should
be able to predict the position of the vehicle in the image. Thus, a detection model with good detection accuracy will
decrease the burden in tracker step and eventually contribute to a higher system accuracy.
The tracker uses Kalman filter with single model (constant velocity) to estimate the vehicle states in each frame.
Constant velocity model is used as most of the vehicle is moving in constant speed within the short road stretch under
monitoring. State vector used is the combination of centroid point, the velocity in x and y directions respectively (vx

Zulaikha et al.
4

and vy), and the width and height of the vehicle box (w and h). In the event of missing observation, Kalman filter is
responsible to predict the possible location of the vehicle respectively.
Track association is completed using the global nearest neighbour based on centroid distance and histogram-based
appearance similarity. Tracker trajectories will be associated with the detected vehicles based on the lowest centroid
and similarity distances.
It is important to terminate and remove the tracker at the end, as the tracker will consume some processing time and
contribute to the overall system performance. Tracker is terminated based on the following few conditions; (1) tracked
object is out of region-of-interest; (2) predicted tracked object box is near or beyond image boundary; (3) tracked
object already counted and exist for more than few seconds after counted; (4) missing observation for more than
certain number of frames.
Figure 3 shows sample images with tracker output information. The green boxes are the output from vehicle detection;
while blue and red are the output from tracker and quantification respectively. Tracker label for each tracked vehicle
is displayed on the top right of the object box. Vehicle trajectories (up to previous 15 frames) are drawn to show the
consistency of the tracker output. Note that even during occlusion, i.e. situation whereby few vehicles are close to
each other, the tracker could still be able to track vehicles correctly to some extend (e.g. frame # 1255).
Figure 3. Sample tracking output; green, blue and red boxes indicate detected vehicle, tracked vehicle, and counted vehicle
respectively. Vehicle will be counted as they pass the line-of-detection (indicated by horizontal black line). Tracker label is
displayed on top of each tracked vehicle box. The tracker trajectories over 15 most recent frames is drawn in black.

2.4Classification
For classification process, we employed Inception v1 model which was released in 2014 by Google Inc. It is also
known as GoogleNet. This network has won the ILSVRC (ImageNet Large Scale Visual Recognition Competition)
in 2014 outperformed other deep learning networks including ZFNet[27], AlexNet[28] and VGGNet[29]. Inception
network architecture has 22 layers and it also has inception blocks in which convolution with different kernel sizes is
done in parallel and the feature map output is concatenated before pass to the next layer. As there exist large gap in
vehicle sizes of different classes (e.g. motorcycle vs. lorry with 3 or more axles), thus, having different kernel sizes as
in inception block might be an advantage to us. Our inception model is constructed by utilizing the pretrained network
weights from [30] which has been trained on ImageNet datasets, followed by a new linear classifier layer which
consists of a linear and a softmax layer with six output correspond to the six vehicle classes as shown in Figure 4. The
linear classifier is then trained using our own datasets, while the other network weights are freezed. The input image
is the RGB images with 224x224 dimension. The data size used during the training is 38352 (90% for the training)
and 4262 (10% for validation) respectively.
The challenges in classification include: (1) interclass correlation is high between class 1 and class 2, and (2) intra
class correlation is low for class 3 and class 4, especially for vehicle of different viewpoints (frontal and rear).

5

Figure 4. Vehicle classes

2.5Quantification
To perform counting, a virtual line is used to trigger the counter to update. As the vehicle crosses the line in the
intended direction, the corresponding vehicle class counter will be increased. Sample snapshot with overlaid virtual
counting line and the direction is shown in Figure 5. In every frame, each of the vehicle centroid will be examined. If
the y-coordinate of the centroid is more than the y-coordinate of the virtual line (assuming the origin of the image is
on the top-left of the image) and the vehicle is not yet counted, then the corresponding class counter will be increased
by one. Counted vehicle will be flagged as “already counted” to avoid double counting on the same vehicle.
For each tracked vehicle, the history of vehicle class prediction will be kept for a small window frame. The final
decision of the counting will be determined based on the highest frequency of the prediction for the tracked vehicle
class.

Figure 5. Sample virtual counting line and the direction.
3.Experimental Setup
3.1Data Collection
Data collection was done using RGB colour video camera, operating with frame rate of 25fps and spatial resolution
(standard) of 704 x 576, storing 24-bit/pixel, mounted on either a lamp post (P) or a pedestrian flyover (F). The camera
set-up for the experiments is illustrated in Figure 6. Proposed camera height is more than 5 meters from the ground
with proposed camera angle is between 45 to 60 degree from the vertical position. This is to ensure that tall vehicles
such as bus and long truck are fully visible in the video, as well as to reduce occlusion level between nearby vehicles.
Sample images of video sequences acquired from different highways are included in Figure 7. Collected data is then
annotated manually by experts in image processing. Summary of these videos properties is specified in Table 1.

Zulaikha et al.
6

Figure 6. Data collection setup
Figure 7. Camera views
Table 1. Summary of collected data
Views Number
of lanes
Vehicle
orientation
Number of
clips
(15min/clip)
Average
number of
vehicles per
clip
1 4 Side (~15°) 12 1134
2 4 Frontal 12 1165
3 5 Side (~45°) 7 2295
4 5 Frontal 8 1593
After the set-up, we noticed good video sequences were obtained with sufficient spatial resolution for the algorithm
to work with but with several challenges: (i) distortion of some video sections by a “shadow effect” caused by the sun
–set and –rise scenario, (ii) image noises appearing at the images corners due to acquisition set-up and highway
structures, (iii) severe occlusion of vehicles especially during peak hours, (iv) motion blur on fast moving vehicle as
shown in Figure 8.

Figure 8. Some challenges observed in the datasets; (a) shadow effects caused by the sun –set –rise and severe occlusion, (b)
motion blur on fast moving vehicle.
On the other hand, initial observation on different vehicle orientation captured from different settings are as follows:
1) for frontal (View 2 and View 4), less occlusion can be observed, but some vehicles from different classes appear
almost the same (e.g. class 1 from class 2 and class 3 from class 4), 2) for side view (View 1 and View 3), it is easier
to differentiate different vehicles especially the trucks with 2 or more axles, however occlusion rate between nearby
vehicles is much higher and become severe during peak hours.
3.2Performance metrics

7

Proposed system is evaluated in terms of (a) vehicle detection rate and accuracy for different detection models; (b)
counting accuracy; (c) counting with classification accuracy; and (d) processing time.
3.2.1Detection Rate and Accuracy
Detection rate and accuracy are calculated by comparing the overlapping region between detected boxes and the
ground truth (GT). For this evaluation, an intercept over union (IoU) of 50% is considered. If a vehicle with the IoU
between its detected box and the ground truth is more than 50%, a true positive (TP) is considered. If the IoU is less
than 50%, it is considered as a false negative (FN) case. While false positive (FP) is considered if the system detects
non-vehicle object as vehicle. In the event where one vehicle is detected with more than one boxes, the first box is
considered as one TP, while the rest are considered as FP.
The detection accuracy (Detacc) and detection rate (DR) are then calculated as follows:
𝑨𝒄𝒄𝒖𝒓𝒂𝒄𝒚, 𝑫𝒆𝒕𝒂𝒄𝒄
𝑻𝑷
𝑻𝑷 𝑭𝑷 𝑭𝑵
∗ 𝟏𝟎𝟎% (1)
𝑫𝒆𝒕𝒆𝒄𝒕𝒊𝒐𝒏 𝒓𝒂𝒕𝒆, 𝑫𝑹
𝑻𝑷
𝑻𝑷 𝑭𝑵
∗ 𝟏𝟎𝟎%
(2)Accuracy,Acc TPTP FP FN*100%Detectionrate,DR TPTP FN*100%

3.2.2Counting Accuracy
Accuracy of the traffic counting is evaluated by finding the error between final system counting output and the GT
for each video clip as in equation 3 and 4. Counting is evaluated per video clip instead of per each frame. As an
example, if a video A has total vehicles of 100 for the duration of 15mins, and the system reports total vehicle of 85,
thus the percentage of error is 15%, and this shall contributes to the accuracy of 85%.
𝐶𝑜𝑢𝑛𝑡𝑖𝑛𝑔 𝑒𝑟𝑟𝑜𝑟,
𝑬𝒓𝒓𝒄 |𝑮𝑻 𝑨𝒄𝒕𝒖𝒂𝒍| 𝐦𝐚𝐱 𝑮𝑻, 𝑨𝒄𝒕𝒖𝒂𝒍
⁄ (3)

𝐶𝑜𝑢𝑛𝑡𝑖𝑛𝑔 𝑎𝑐𝑐𝑢𝑟𝑎𝑐𝑦,
𝑨𝒄𝒄𝒄 𝟏 𝑬𝒕𝒄 𝒙𝟏𝟎𝟎% (4)
3.2.3Counting with Classification Accuracy
For traffic counting with classification, the overall accuracy is calculated using equation (5), where it is the weighted
average of the traffic counting accuracy for each class calculated using equations (4). The weight is determined based
on the ratio of GT number of each vehicle class to total vehicle in the video clip as in equation (6).
𝐶𝑜𝑢𝑛𝑡𝑖𝑛𝑔 𝑤𝑖𝑡ℎ 𝑐𝑙𝑎𝑠𝑠𝑖𝑓𝑖𝑐𝑎𝑡𝑖𝑜𝑛 𝑎𝑐𝑐𝑢𝑟𝑎𝑐𝑦,
𝑨𝒄𝒄𝒄𝒄 ∑ 𝒘𝒄𝒊 ∗ 𝑨𝒄𝒄𝒄𝒊
𝟔
𝒊 𝟏 (5)

𝑉𝑒ℎ𝑖𝑐𝑙𝑒 𝑐𝑙𝑎𝑠𝑠 𝑥 𝑤𝑒𝑖𝑔ℎ𝑡,
𝒘𝒄𝒙
𝑮𝑻𝒄𝒙
∑ 𝑮𝑻𝒄𝒊
𝟔
𝒊 𝟏
(6)
4.Results and Discussion
4.1Detection rate and accuracy
For evaluating detection accuracy and detection rate, three 20-seconds video clips (~500 image frames) are selected
from three camera views. As highlighted in Table 2, these videos are carefully chosen to emphasize different problems

Zulaikha et al.
8

occurring in complex traffic scene, such as (1) high variance of object size (e.g. motorcycle vs lorry with more than 2
axles) (2) high occlusion between nearby objects (e.g. during peak hour); and (3) different vehicle orientations (e.g.
frontal vs side view). For side view, the angle given is the angle between vehicle’s major axis and the vertical line.
The threshold for minimum confidence value is set to 0.3 for both detection models.
Table 2. Testing video properties
Dataset
(40sec
videos)
Total
vehicles
Min and
max object
size (%)
Occlusion
level
Orientation
Video 1 1522 0.37, 17.86 Medium Side (~15°)
Video 2 1454 0.23, 18.45 Low Frontal
Video 3 5015 0.09, 12.45 High Side (~45°)
These datasets have been manually labelled whereby the vehicle type and bounding box location of each vehicles
appears at least 80% of the body within the detection area is recorded as the ground truth.
Table 3 shows the detection accuracy and detection rate of the proposed detection models on the 3 selected video clips.
YOLO model consistently achieved the highest accuracy and highest detection rate in all videos. YOLO achieves the
highest accuracy for Video 3 despite of occlusion level in the video, may due to the orientation of the vehicle which
is about 45° from the vertical line. The vehicle appearance is much more distinguishable from this view as compared
to frontal view. This is proved by the lowest accuracy achieved by both models in Video 2.
Figure 9 illustrates sample of detection results from Video 3. Green and blue boxes are the detected vehicles in and
out of detection area respectively. While the red boxes are the miss detected vehicles. These samples show that YOLO
works great in detecting small and occluded vehicles.
In terms of processing time, SSD runs faster compared to YOLO with average inference time of 9.82 ms/frames, while
YOLO runs at an average of 17.88 ms/frames as listed in Table 4. Both models are running on Nvidia GeForce RTX
2070, intel® Core™ i5-8400 CPU@2.80GHz. Based on the testing results, YOLO is chosen as the model for our
system as it gives the best results in terms of accuracy and detection rate and able to run in real-time.
Table 3. Detection rate and accuracy result on selected 40 seconds videos
Dataset
(40sec
videos)
Models TP FP FN DETACC
(IOU =
50)
DR(%)
Video 1 SSD 996 11 456 68.12 68.64
YOLO 1161 0 296 79.64 79.64
Video 2 SSD 668 10 865 43.29 43.57
YOLO 1139 7 383 74.49 74.84
Video 3 SSD 3266 24 1750 64.80 65.11
YOLO 4801 68 214 94.45 95.73

9

Figure 9. Sample vehicle detection result using different detection models. Green, blue and red boxes are the detected vehicle in
detection area, detected vehicle out of detection area and miss detected vehicle respectively.
Table 4. Average inference time for SSD and YOLO
Detection
model
Average inference time
(ms)
Average inference time
(fps)
SSD 9.82 101.83
YOLO 17.88 55.92
4.2Counting Accuracy
Figure 10 shows the box plot of the accuracy from all four views. Based on the results in Figure 10, overall, the system
has achieved good accuracy results for all video clips, with minimum accuracy of 92.89% is recorded from view 3 for
clip between time 1300 to 1315. View 2 achieves the highest mean accuracy, followed by view 1 and view 4, while
view 3 records the lowest overall mean accuracy.
Although detection accuracy of samples from view 3 is higher compared to the other views, but the occlusion level is
also higher. The occlusion may not impact much on the detection process, however, the tracker may have problem in
associating tracker trajectories with the detected vehicles when they are closed and occluded to each other. That gives
impact to the overall counting accuracy. Nevertheless, it is also observed that overall counting accuracy is high despite
of low detection rate, and this shows that the tracker is able to maintain same vehicle identity during missing
observation from detection step.

Figure 10. Box plot of counting results from four camera views
4.3Counting with Classification Accuracy
Each of the counting results is further evaluated according to their vehicle classification output. Figure 11 – Figure 14
depict the accuracy of each class (bar chart) and the weighted average accuracy (line chart) for each camera view
respectively. View 1, view 2 and view 3 record weighted average accuracy above 90% for all video clips. However,
view 3 records lower weighted average accuracy in general, with the lowest is logged at 70.48% for the clip between
1600-1615 hours. By looking at Figure , lower overall accuracy is contributed by the low-class accuracy in C2, C4
and C6.

Zulaikha et al.
10

Figure 11. Weighted average accuracy for View 1



11

4.4Processing Time
The system has been implemented in C++, with inference engine of TensorRT [31]. All model was converted
to TensorRT optimized models for deployment. The system is running on the Nvidia GeForce RTX 2070, intel®
Core™ i5-8400 CPU@2.80GHz. Average processing time for each process is shown in
Table 5. Overall, the system achieves real-time processing with average of 26.98 frame per second (fps), which is
higher than the frame rate of video capturing process at 25fps. Detection step requires longer processing time
compared to other processes.
Table 5. Average processing time (per frame)
Detection
(ms)
Tracker
(ms)
Classification
(ms)
Quantification
(ms)
Overall
processing
Time
(ms)
Fps
17.8 9.70 9.49 0.00094 37.06 26.98
5.Conclusion and Future Work
A practical framework is proposed in this paper to deploy a real-time vehicle counting system, using (1) an end-to-
end systematic traffic flow estimation system in urban setting; (2) a multi-level CNN that could handle vehicles from
different camera views; and (3) various dataset to validate the effectiveness of the proposed vehicle counting
technique. Experimental results show that the mCNN could track and count vehicles under various conditions,
achieving average counting accuracy and average weighted counting with classification accuracy of 97.53% and
90.2% respectively for all 39 video clips of duration 15-mins each.
From the detection evaluation, it is shown that YOLO outperformed SSD in all sample videos with the highest
accuracy is recorded for vehicles from side view (about 45degree angle between vehicle’s major axis and vertical line)
as the vehicle appearance is more distinguishable from this view. However, occlusion level is also higher in this view.
The occlusion level impacts the tracker’s performance, which cause the overall counting accuracy of this view is lower
than the others.
In addition to accuracy, real time processing is also a crucial factor in the system deployment and it is proved that our
mCNN implementation executes in real-time as the average processes is faster than 25 fps of camera capturing rate
with average overall processing of our system is 37.06ms per image, which equivalent to 26.98fps.
Acknowledgement
None

Zulaikha et al.
12

Conflicts of interest
The authors have no conflicts of interest to declare.
References
[1] Nicholas A. Mandellos, Iphigenia Keramitsoglou, Chris T. Kiranoudis. A background subtraction algorithm for detecting and
tracking vehicles. Expert Systems with Applications. 2011; 38(3):1619-1631.
[2] Siu-Yeung Cho, Chai Quek, Shao-Xiong Seah, Chin-Hui Chong. HebbR2-Taffic: A novel application of neuro-fuzzy network
for visual based traffic monitoring system. Expert Systems with Applications. 2009; 36(3):6343-6356.
[3] Y. Liu, B. Tian, S. Chen, F. Zhu and K. Wang. A survey of vision-based vehicle detection and tracking techniques in ITS.
IEEE Int. Conf. on Vehicular Electronics and Safety (ICVES). 2013:72-77.
[4] Z.-J. Zha, M. Wang, Y.-T. Zheng, Y. Yang, R. Hong, T.-S. Chua. Interactive video indexing with statistical active learning.
IEEE Trans. Multimedia. 2012:17–27.
[5] Sobral. Vehicle Detection, Tracking and Counting. 2014. https://github.com/andrewssobral/simple_vehicle_counting
[6] X. Liu, Z. Wang, J. Feng, H. Xi. Highway vehicle counting in compressed domain. IEEE Conference on Computer Vision
and Pattern Recognition. 2016:3016–3024.
[7] Y. Xia, X. Shi, G. Song, Q. Geng, Y. Liu. Towards improving quality of video-based vehicle counting method for traffic flow
estimation. Signal Process. 2016; 120:672–681.
[8] R, Vikram. Density based Traffic Management System using Image Processing. International Journal of Advanced Trends in
Computer Science and Engineering. 2020; 9:997-1001.
[9] Lai, J.-C., Huang, S.-S., & Tseng, C.-C. Image-based vehicle tracking and classification on the highway. IEEE International
conference on green circuits and systems (ICGCS). 2010:666–670.
[10] Shen, Y. The research and applications of visual tracking based on meanshift (Ph.D. thesis). Jiangsu University of Science
and Technology. 2008.
[11] Takeuchi, A., Mita, S., & McAllester, D. On-road vehicle tracking using deformable object model and particle filter with
integrated likelihoods. IEEE intelligent vehicles symposium (IV). 2010:1014–1021.
[12] Tu, Q., Xu, Y., & Zhou, M. Robust vehicle tracking based on scale invariant feature transform. IEEE International conference
on information and automation (ICIA 2008). 2008: 86–90).
[13] Bouttefroy, P. L. M., Bouzerdoum, A., Phung, S. L., Beghdadi, A. Vehicle tracking by non-drifting mean-shift using projective
kalman filter. 11th
International IEEE conference on intelligent transportation systems (ITSC 2008). 2008:61–66.
[14] Xie, L., Zhu, G., Wang, Y., Xu, H., & Zhang, Z. Real-time vehicles tracking based on Kalman filter in a video-based ITS.
2005 International conference on communications, circuits and systems. 2005; (2).
[15] Scharcanski, J., de Oliveira, A. B., Cavalcanti, P. G., & Yari, Y. A particle filtering approach for vehicular tracking adaptive
to occlusions. IEEE Transactions on Vehicular Technology. 2011; 60:381–389.
[16] Ren, S.; He, K.; Girshick, R.; Sun, J. Faster R-CNN: Towards Real-Time Object Detection with Region Proposal
Networks. IEEE Trans. Pattern Anal. Mach. Intell. 2017; 39:1137–1149.
[17] Liu, W.; Anguelov, D.; Erhan, D.; Szegedy, C.; Reed, S.; Fu, C.Y.; Berg, A.C. SSD: Single Shot MultiBox Detector. In
Proceedings of the Computer Vision—ECCV. 2016:21–37.
[18] Redmon, J.; Divvala, S.; Girshick, R.; Farhadi, A. You Only Look Once: Unified, Real-Time Object Detection. In Proceedings
of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2016.
[19] Z. Kadim, K. M. Johari, D. F. Samaon, Y. S. Li and H. W. Hon. Real-Time Deep-Learning Based Traffic Volume Count for
High-Traffic Urban Arterial Roads. IEEE 10th Symposium on Computer Applications & Industrial Electronics (ISCAIE).
2020:53-58.
[20] Nazirah Zulkifli, Muhammad Amirul Idham Laili, Siti Nurshaline Mohd Saadon, Adizul Ahmad, Azlee Zabidi, Ihsan Mohd
Yassin, Fadhlan Hafizhelmi Kamaru Zaman Megat Syahirul Amin Megat Ali, Mohd Nasir Taib. Fast Region Convolutional
Neural Network Lane and Road Defect Detection for Autonomous Vehicle Application. International Journal of Advanced
Trends in Computer Science and Engineering. 2019; 8.
[21] Ross Girshick, Jeff Donahue, Trevor Darrell and Jitendra Malik. Rich feature hierarchies for accurate object detection and
semantic segmentation. IEEE Conference on Computer Vision and Pattern Recognition. 2014:580-587.
[22] Ross Girshick. Fast R-CNN. IEEE International Conference on Computer Vision (ICCV). 2015:1440-1448.
[23] Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng-Yang Fu, Alexander C. Berg. SSD:
Single Shot MultiBox Detector. Github. https://github.com/weiliu89/caffe/tree/ssd. Accessed 30 Jan 2020.
[24] Lin TY. et al. Microsoft COCO: Common Objects in Context. In: Fleet D., Pajdla T., Schiele B., Tuytelaars T. (eds)
Computer Vision – ECCV 2014. Lecture Notes in Computer Science. 2014; 8693:740-755.
[25] J. Redmon and A. Farhadi. Yolov3: An incremental improvement. arXiv. 2018.
[26] J. Redmon and A. Farhadi. Yolov3 weight file. https://pjreddie.com/media/files/yolov3.weights. Accessed 30 July 2020.
[27] Zeiler M.D., Fergus R. Visualizing and Understanding Convolutional Networks. In: Fleet D., Pajdla T., Schiele B., Tuytelaars
T. (eds) Computer Vision – ECCV 2014. Lecture Notes in Computer Science. Springer, Cham. 2014; 8689: 818-833.
[28] Krizhevsky, Alex & Sutskever, Ilya & Hinton, Geoffrey. ImageNet Classification with Deep Convolutional Neural Networks.
Neural Information Processing Systems. 2012; 25(2).

13

[29] Simonyan, Karen & Zisserman, Andrew. Very Deep Convolutional Networks for Large-Scale Image Recognition. arXiv
1409.1556. 2014.
[30] GoogLeNet. https://www.dropbox.com/sh/axnbpd1oe92aoyd/AADpmuFIJTtxS7zkL_LZrROLa?dl=0. Accessed 30 Mar
2020.
[31] NVIDIA Developer. NVIDIA TensorRT programmable inference accelerator. https://developer.nvidia.com/tensorrt.
Accessed 30 July 2020.
Zulaikha Kadim received her degree and master degree in Engineering from Multimedia University
Malaysia. Currently she is pursuing her PhD in Computer Systems Engineering at Malaysia National
University (UKM). She is currently a senior researcher in MIMOS Berhad. Her research interests include
object detection & tracking, and video analytics.
Email:zulaikha.kadim@mimos.my
Khairunnisa Mohamed Johari received her B.Science (Intelligent System) from Mara University of
Technology (UiTM) in 2010. She is currently a senior researcher in Advanced Informatics Lab, MIMOS
Berhad. Her current research interest are in video analytics, deep learning and behavioral analysis.
Email:nisa.johari@mimos.my
Den Fairol Samaon received his B.Science (Computer Science) from University f Technology
Malaysia(UTM) in 2003. He is currently a researcher in Advanced Informatics Lab, MIMOS Berhad. His
current research interest are in computer vision, deep learning and object detection.
Email:fairol.samaon@mimos.my
Yuen Shang Li received Bachelor of Science in Electrical and Electronic Engineering in 2009 and Master
Degree of Engineering in Electrical Engineering, majored in Automation and Robotics System in 2011.
Currently he is working as research engineer in national R&D institution. His current research interests are
in deep learning, video analytic, action recognition and license plate recognition.
Email:sl.yuen@mimos.my
Hon Hock Woon received his bachelor degree in Electrical and Electronic Engineering from the Nottingham
Trent University, United Kingdom in 1997 and his doctorate degree in the same university in year 2000. His
main research areas are in imaging/image processing, computer vision and machine learning. Dr. Hon
currently holds a position of senior principal researcher at Mimos Berhad.
Email:hockwoon.hon@mimos.my

Real time vehicle counting in complex scene for traffic flow estimation using multi-level convolutional neural network

More Related Content

What's hot (20)

Similar to Real time vehicle counting in complex scene for traffic flow estimation using multi-level convolutional neural network (20)

More from Conference Papers (20)

Recently uploaded (20)

Real time vehicle counting in complex scene for traffic flow estimation using multi-level convolutional neural network