Ridesharing Passengers and Driver Safety Using Emotion Recognition and Weapons Detection Systems

1.1 Review of related studies

This chapter implements image processing techniques to improve both driver and passenger safety of ride-share trips. Image processing for driver safety is a long-standing research domain and has seen outstanding results on lab-controlled data and real-life scenarios; this is the case presented by Xiao et al. [1]. Driver safety has been implemented using image processing by executing a drowsy driver detection, which detects when a driver might be drowsy and plays a song or alerts an alarm to wake the driver up, preventing accidents and promoting road safety. Other studies by Xiao et al. and Sukhavasi et al. [1, 2] have introduced emotion recognition to analyze the mood and mental state of the driver, have seen considerable results, and have allowed them to put measures in place to address drivers who are too tired or not emotionally ready to drive. However, extending these techniques to ensure passenger safety remains an underexplored domain; therefore, seeing the technology results and success, this study aims to introduce image processing for the driver and the passenger.

It is no secret that ride-share/pooling companies have encountered a surge in harassment complaints and instances of gender-based violence against passengers, as reported by Mlamla and Bonyhady [3, 4]. Not so long ago, in March 2022, Mlamla 2022 reports, a petition signed by over 90,000 people circulated requesting “Bolt to vet its drivers more efficiently” [3]. In November 2021, an article in The Sydney Morning Herald by Bonyhady 2021 reported over 6 months, Uber received over 500 sexual misconduct and assault complaints from passengers and drivers [4]. It is a concern that sadly happens across the globe, and more solutions (besides panic buttons) are needed.

Currently, Uber has launched the following safety measures, “Share My Trip,” “in-app Emergency Button,” “24/7 customer support,” “safety centre,” and checking “Driver Profile” before stepping into the car and taking the trip. These are all excellent features, but they come with some difficulties and may be challenging to use in a distressed situation. According to Mohan [5], 48 of 50 rides taken by the Indian Express did not have the panic button, or it simply did not work. In distressed situations, it can also be difficult for the passenger to press the emergency button, seeing that it is in the app, and the passenger could be threatened or held at gunpoint, leaving them helpless and unable to use the emergency button, same applies to the in-app safety centre and 24/7 customer support. Additionally, drivers are exposed to the risk of theft or attacks, as some drivers accept cash trips. With the public aware of this, theft is a great concern, and more measures should be put in place to combat this because drivers accumulate money throughout the day from cash trips.

Chaudhry et al. pointed out that the progress and growth of ridesharing companies could depend on the customer’s safety and security [6]. Security problems with ridesharing rides are not just in one town or country; these problems grow with the service’s popularity. To improve the security of these trips, Chaudhry et al. suggested the following solutions, “Dash Cam and Watchdog Network,” “Distress Alarm,” “Passenger Insurance,” and other “Miscellaneous” [6]. The suggested solution involves a dashcam that records the trip and makes the live footage accessible to a watchdog network and the passengers’ social media profiles to have more eyes on the trip. The distress alarm has recently been implemented in the form of panic buttons.

Facial Emotion Recognition (FER) consists of three steps: face detection, facial expression detection, and classifying the expression to an emotional state. Vemou and Horvath, state that FER takes static images and videos as inputs to analyze facial expressions [7]. Without a doubt, I am not the first to use FER, as FER is a growing technology used in various fields. FER has been used in healthcare, employment, education, and retail, analyzing customer behavior, to name a few.

Mehendale, presented a novel approach for FER, using “two-level” CNN, where the first level removes the background from the picture/video, and the second focuses on extracting the facial feature vectors [8]. However, as highlighted by Lim and Teo 2020, FER is not limited to image processing; eye-tracking can be used for the same purposes [9].

Weapon detection is a task that can be performed using various techniques and approaches [10]. Developing an automatic detection system that can detect potentially dangerous situations ensures effective prevention and the security of civilians; one way of detecting a potentially dangerous situation is by detecting weapons such as handguns [10]. Ruiz-Santaquiteria et al. implemented a novel approach for handgun detection [11]. They combined two different models with different purposes, the one meant to differentiate recognizable body poses and the other for hand image region classifications. The results of the models are then fed to an optional CNN-based model for filtering false positives [11]. The study utilized OpenPose for the body pose estimation, generated a numerical feature matrix with two-dimensional key points, normalized the key points, tested different models, and used the Darknet-53 extractor from YOLOv3 [11]. The dataset was a combination of datasets and images obtained from various sources. The freely available datasets include the Guns Movies and COCO datasets to balance out the overall dataset, with images excluding handguns [11]. Other data was collected from YouTube and video game watchdogs to mimic CCTV images. The following metrics were used for model evaluation and comparison: Precision, A.P. score, and Recall. In the end, they successfully improved the results of handgun detection by introducing body poses.

Weapon detection was used by Wang et al. for security and to fight terrorism [12]. The study utilized the YOLOv4 model, which improves on the drawbacks of the previous versions. These drawbacks include poor positioning accuracy and the inability to pick up small, dense objects [12]. They then further improved the YOLOv4 to detect smaller objects better. Improvement was achieved by implementing an SCSP-ResNet with an improved Spatial Attention Mechanism and a receptive field enhancement module with a GoogleNet architecture [12]. The dataset comprises synthetic and actual data generated by Unity Game Engine, including a real-world attack simulation and a freely available dataset [12]. Experiments saw the comparison of the improved YOLOv4 model with the original model, the other YOLO algorithms, and other CNN models. Evaluation metrics used were detection precision, true positives and false positives, recall, accuracy, and F1 score [12].

Datasets from the reviewed studies were created or obtained through online security footage. Videos and images went through preprocessing, in which all background and noise were removed, and the appropriate features were extracted. This is then used to train the model to identify the different levels of danger, and the model will then be able to classify and detect any threats or possible dangers.

If one successfully implements FER in ridesharing vehicles, an excellent place to start is to implement an open-source and lightweight model, as these models are cost-effective and require less powerful hardware, as such hardware would not necessarily be found in a ridesharing vehicle or any vehicle. Sampaio et al. cover several such models/libraries and their performance on a sample of the FER2013 dataset used in this study [13]. The models covered are DeepFace, EmoPy, Py-FEAT, and residual mask network (RMN). According to Sampaio et al. DeepFace is a lightweight Python library that provides diverse models tailored for facial recognition and attribute analysis [13]. Among those it offers is a pre-trained CNN trained on the FER-2013 dataset, achieving a 57% accuracy [13]. EmoPy is another open-source Python library providing various pre-trained neural network architectures for FER projects [14]. Py-FEAT, an open-source Python package, offers tools for face detection and extracting facial expressions, muscle movements, and landmarks from videos and images [13]. RMN is the last Python library covered and offers a pre-trained residual mask network (RMN) for the prediction of emotions [13]. The metrics in focus are “Accuracy” (A), “Precision” (P), “Recall” (R), “Cohen’s Kappa” (K), and “F1-score” (F). From the results in the article, it is seen that DeepFace outperformed across all metrics; therefore, the DeepFace results are used to compare with the results of this study.

Over the years, YOLO models have become quite popular and essential in the real-time detection of objects, with a rollout of a new model to deal with the limitations of predeceasing versions happening every now and so often. At the time of writing, the latest version of the model was YOLOv8; considering that, Terven and Cordova-Esparza, performed a comprehensive review study of the various YOLO models [14]. The study highlights the changes made to the model, such as new network backbones, a better technique to augment data, and different optimization techniques. The study also points out that from YOLOv4, there were slight improvements in model accuracy without affecting the real-time performance of the models but with an increase in computational costs. Therefore, YOLOv4 is used in this work.

In a research study to develop and optimize a weapon detection system, Ahmed et al. sought to improve real-time weapon detection and explored large-scale deployment [15]. The study highlights key areas of large-scale deployment and challenges that arise from cloud deployment hardware limitations and uses that information not only to improve the accuracy and performance of existing weapon detection models and improve the number of frames per second for real-time deployment but also to compare the performance of the models on different computing devices. The methodology emphasizes the importance of datasets and how they affect algorithms. The dataset used for training and testing consisted of 8327 labeled images split into training and testing datasets. To improve the mAP model, the datasets were broken into two categories, pistol and non-pistol, which helped reduce false negatives and false positives. The model used was YOLOv4 and TensorRT to optimize the model into YOLOv4-Scaled for network optimization in edge-computing with comparison results favoring the YOLOv4-Scaled model. In future works, Ahmed et al. mention using body poses to improve the system [15]. This article has aided my research project by giving me insight into developing systems for large-scale deployment optimization techniques and a successful systematic approach to developing a weapon detection system using YOLO.

1.2 State-of-the-art review

This review discusses the state-of-the-art technologies in emotion recognition and weapons detection systems, examining their applications in enhancing safety for ridesharing passengers and drivers.

2.1 Data acquisition

Input sources: Cameras capture real-time video streams of the vehicle’s interior.

Data types: Image Data - Captures facial expressions and objects.

2.2 Preprocessing

Preprocessing prepares raw data for analysis by improving its quality and suitability for recognition algorithms.

Image preprocessing: Convert video frames to grayscale. Resize images to match model input dimensions (e.g. 48 × 48 pixels for FER2013-based models). Normalize pixel values (e.g. scale from 0 to 255 to 0–1) to improve computational efficiency.

Face detection: Apply algorithms like Haar Cascades or deep learning-based methods (e.g. Multi-task Cascaded Convolutional Networks (MTCNN)) to isolate faces from the video frames for further emotion analysis.

Object detection preparation: Use bounding box algorithms to focus on potential regions containing objects for weapon detection.

2.3 Feature extraction

Feature extraction identifies and highlights relevant patterns in the data, which are then fed into machine learning models.

Facial features for emotion recognition: Extract landmarks (e.g. eye, nose, and mouth positions) to aid in identifying facial expressions. Input pixel values into CNN layers to identify high-level features corresponding to emotions.

Object features for weapon detection: Input object region data into YOLOv4 or similar models to identify characteristic shapes and textures of weapons.

2.4 Model processing

This stage involves using trained machine learning models to process the extracted features and generate outputs.

Emotion recognition: CNNs: Convolutional Neural Networks process image data through layers of filters, pooling, and fully connected layers to classify emotions (e.g. angry, fear, and happy).

Weapon detection: YOLOv4 - Processes regions of interest to detect objects in the frame, assigning probabilities to classify them as weapons or non-weapons.

2.5 Decision making

After processing, the system interprets the model’s outputs to make safety-related decisions.

Emotion analysis: High-confidence detection of negative emotions (e.g. fear and anger) triggers an alert system.

Weapon detection: Positive identification of a weapon triggers a higher-priority alert, which may involve emergency escalation.

2.6 Alert system

The processed results are communicated to relevant stakeholders via a user interface.

Visualization: The Streamlit GUI displays real-time results, such as the detected emotion (“Fear”) or object (“Handgun”).

Notifications: If a safety risk is detected (e.g. a weapon or heightened emotion), the system sends alerts to the driver and passenger (in-app notifications), and the ridesharing company’s support or emergency team.

This chapter presents a two-part danger detection collaborative safety tool: a GUI web app that ride-share companies like Uber and Bolt can use during trips. The two detection methods employed are emotion detection and handgun detection. These two detection methods are integrated using a Streamlit GUI. An alarm is triggered upon detecting the respective threats, and an image is saved. This section explains the integrated threat identification and response approach. Figure 1 is a flowchart outlining the important stages of the overall project.

Figure 1 presents an overview of the research design and methodology for integrating emotion recognition and weapons detection systems in a ridesharing safety framework. The design includes two main detection pathways—one focused on weapons (e.g. handguns) and the other on emotional states (e.g. fear). These options are outlined as follows:

Object detection and face detection: Both pathways begin with object detection and face detection processes. Object detection identifies items within the vehicle environment, while face detection isolates face for emotional analysis.

Handgun recognition (YOLOv4): For weapon detection, YOLOv4 (you only look once version 4) is used to recognize handguns specifically. This model is selected for its real-time detection capabilities, allowing the system to alert operators immediately if a weapon is identified.

Emotion recognition (fear) using sequential CNN: For emotion detection, a sequential convolutional neural network (CNN) is employed to recognize specific emotional cues, particularly fear. This model processes facial expressions to identify emotional states that could indicate distress or potential conflict.

Streamlit GUI and alert system: Both detection pathways feed into a Streamlit GUI, which serves as a user interface, displaying real-time outputs from the detection systems. If either a weapon or an emotional cue like fear is detected, an alert system is triggered to notify the ridesharing support team, enabling prompt intervention.

This dual-path approach allows the system to detect both physical threats and high-risk emotional states, enhancing ridesharing safety through a layered, proactive monitoring solution (Figure 1).

2.7 Dataset collection

For this chapter, both datasets are open data found on the Kaggle platform. For training and testing the Emotion Recognition model, the FER2013 dataset introduced by Dumitru et al. was utilized [26]. In contrast, the “Handgun Detection” dataset provided by the University of Granada [10] was used for the Handgun detection model. Exploration and preprocessing were conducted on the dataset.

2.7.1 FER2013 dataset

The FER2013 dataset (Facial Expression Recognition 2013) is a widely used benchmark for emotion recognition tasks, particularly in applications involving computer vision and machine learning. It was originally introduced for the ICML (International Conference on Machine Learning) 2013 competition and has since become a standard dataset for training and evaluating models designed to recognize human emotions through facial expressions.

The FER2013 Dataset consists of 32,568, 48 × 48 grayscale images of faces displaying seven different emotions, that is, anger, disgust, fear, happiness, neutrality, sadness, and surprise which is described in the Figure 1 as Option 2 (Emotion). From the 32,568 images, the dataset is further split into 28,709 Train images and 3859 Test images.

FER2013 is extensively used for training convolutional neural networks (CNNs) in facial expression recognition tasks. The dataset is particularly valuable for deep learning models like CNNs due to its size and diversity of facial expressions. Many modern emotion recognition systems, including those for customer service, healthcare, and security applications, use FER2013 to develop real-time recognition capabilities.

For ridesharing safety applications, FER2013 can be instrumental in training models to detect emotions like “Fear” or “Angry,” which are associated with potential risks. By recognizing these emotions in real time, systems can alert operators or support teams to de-escalate situations before conflicts arise. Its range of emotions makes FER2013 particularly useful for recognizing distress or aggression in confined environments, enhancing safety monitoring in scenarios like ridesharing or public transportation.

By exploring the dataset folders, it was noticed that both the train and test sets were imbalanced, as seen in Figure 2. Using imbalanced datasets to train models is not good practice as it introduces a bias towards the dominant class in the dataset.

This study implemented data augmentation to balance the datasets and reduce some folders. In addition to balancing out the datasets, sampling was also introduced. Sampling uses a fraction of the total dataset as opposed to the entire dataset due to the high computation costs of using the whole dataset. The final balanced datasets comprised 28,000 training images and 4655 testing images, using an 85/15 train-test split. Each training emotion has 4000 images, while each test emotion hosts 665 images.

The balanced datasets were then normalized by dividing each pixel value by the max pixel value, 255. Normalization is a crucial preprocessing step as it ensures all inputs follow a standard distribution, ensuring the gradient calculations computed in the model network remain consistent, helping the model train better and faster. The final preprocessing step was resizing the images to match the required input size for each model.

2.8 Modeling

To achieve robust facial emotion recognition, this study trained/used and compared three distinct models: the pre-trained model offered by the lightweight DeepFace python library, a sequential CNN, and MobileNetv2 using transfer learning. The selection of DeepFace was due to it outperforming its lightweight counterparts, as seen in [13], as well as it being a lightweight open-source library for facial recognition comprising several state-of-the-art AI models specialized in facial recognition, such as VGG-Face, Facenet, OpenFace, DeepFace, Dlib, and ArcFace. Notable lightweight absentees are the Sequential and MobileNetv2 models; hence, they were the other models trained and tested in this study. Additionally, utilizing the YOLOv4 model is a cornerstone for object detection. Its cutting-edge architecture and capabilities are ideal for detecting objects, including handguns.

2.8.1 Facial emotion recognition

Figure 3 gives a brief overview of the critical stages in the facial emotion recognition model in the form of a flow chart. The key stages are further discussed below.

Figure 3 provides a concise overview of the facial emotion recognition process, which involves the following sequential steps:

1. Image acquisition: This is the initial step where images are captured using cameras installed in the ridesharing vehicle or other monitoring systems. These images serve as the raw data for processing. Live image footage of different facial emotions was acquired in this study via the laptop webcam.

2. Face detection: The system isolates faces from the captured images using algorithms like Haar Cascades, Multi-task Cascaded Convolutional Networks (MTCNN), or other deep learning-based methods. This step focuses only on the regions of interest (the faces) while ignoring other parts of the image.

3. Face recognition: After detecting a face, the system analyzes it to identify specific emotional features. Convolutional Neural Networks (CNNs) or similar machine learning models are typically employed here to classify the emotion (e.g. anger, fear, and happiness) based on the detected facial features.

4. Safe and alert: Depending on the recognized emotion, the system decides. The image or video frame containing the detected handgun is stored in a secure database for record-keeping and potential evidence. Also, an immediate alert is triggered to notify relevant stakeholders, such as the ridesharing company’s security team or law enforcement, ensuring timely intervention.

Before any facial emotions are detected and inferences are made, facial detection is done on the acquired image to ensure a face. This study used the OpenCV Haar-cascade classifier for face detection by downloading and loading it into the haarcascade_frontalface_default.xml file. Haar Cascades is an object detection algorithm for finding faces in pictures or real-time video analysis. The algorithm uses edge and line detection and works by using “positive images” with faces and “negative images “that do not have faces. According to Behera 2020, the algorithm trains these positive and negative images to identify if images have a face in them [28]. After the face is detected, a bounding box is drawn around the face.

2.8.1.1 DeepFace

Since DeepFace is a pre-trained model, all that was done to utilize it was to import the DeepFace library from the deepface package. DeepFace has an inbuilt function called “analyze” which offers information on the detected face from the acquired image, such as the different possible emotions as well as the dominant emotion, age, gender, and various possible races as well as the dominant race, an example of this can be seen in Figure 4. This study focuses only on the predicted dominant emotion, which is then printed in the bounding box created during the face detection step.

Figure 4 illustrates the functionality of the DeepFace framework for analyzing facial data to recognize emotions. DeepFace is an advanced facial analysis tool that utilizes deep learning to process and interpret facial features.

Face detection: The system first identifies and isolates faces from the input image or video stream. This step ensures that only relevant facial regions are analyzed.

Feature extraction: Using a deep learning model, DeepFace extracts key facial features such as the eyes, nose, mouth, and other distinguishing elements. These features are converted into numerical representations (embeddings) for processing.

Emotion recognition: The extracted features are processed to classify the individual’s emotional state. DeepFace typically recognizes a range of emotions, such as happiness, sadness, anger, surprise, and fear, using pre-trained neural networks.

Output generation: The results of the analysis are displayed, indicating the detected emotion(s) with corresponding confidence levels. These outputs can be visualized through a graphical user interface or logged for further analysis.

2.8.1.2 MobileNetV2

MobileNetV2 is a lightweight convolutional neural network model built for image classification. It aims to excel in performance when deployed on mobile and edge-computing devices; hence, it was selected. It adopts an inverted residual structure, employing residual connections within bottleneck layers. According to Sandler et al. the intermediate expansion layer utilizes lightweight depth-wise convolutions to filter features, providing non-linearity [29]. The model is downloaded from tensorflow keras.

Keras has a pre-trained MobileNetV2 model; therefore, transfer learning was implemented in this study. Transfer learning refers to leveraging knowledge/features gained from one task and applying it to a different but related task [30]. MobileNetV2 is said to be trained “on more than a million images from the ImageNet database” [31]. Due to the limitation on the computation resources, MobileNetV2 only allowed 1400 images to be trained. Therefore, the dataset was reduced to 1400 train images (200 per emotion) and 350 test images (50 per emotion).

MobileNetV2 takes the input of image size 224 × 224, 3. Therefore, the images were resized to fit this required input size.

Several new layers were added to the pre-existing model; two dense layers with 128 and 64 units perform a linear operation on the input from the previous layers, using ReLU activation functions, which introduce non-linearity. Dropout layers with a dropout rate of 0.2 after each dense layer to prevent overfitting. A final dense layer with seven units since it is a classification task with seven classes (seven emotions) using a softmax activation function to output probabilities for each class. Softmax was chosen as it is suitable for a multiclass problem.

Figure 5a and b shows the training of the MobileNetV2 model. The model achieved high training accuracy, but as seen by the validation accuracy line, overfitting occurred despite the dropout layers as the validation accuracy continuously decreased (Figure 5a). Figure 5b illustrates the training and validation performance of the MobileNetV2 model during the training process. This comparison highlights how well the model generalizes to unseen data after being trained on a specific dataset.

2.8.1.3 Sequential CNN

The model takes an input of size 48 × 48 with three channels, which is already the size of our dataset; however, images were resized to the desired input to prevent any unnoticed incorrect input size.

The architecture employed to build the Sequential model:

• Conv2D layer: Constructs a convolutional layer with a given number of filters (32, 64, or 128), kernel size (3, 3), ReLU activation function, ‘same’ padding to preserve spatial dimensions, and an input shape for the first layer. The filters increase from 32 to 64 to 128, gradually capturing more complex features.

• BatchNormalization layer: Normalizes the previous layer’s activations, allowing faster convergence during training and minimizing overfitting.

• MaxPooling2D layer: Down samples the spatial dimensions by taking the maximum value inside a defined window (2 × 2).

• Dropout layer: The dropout layer prevents overfitting by setting a certain percentage of the input to 0, preventing the model from memorizing the data and enabling learning patterns. In this case, 25% of the input is set to 0.

• Flatten layer: Flattens the previous layers’ output into a 1D array, making it ready to input the dense layers.

• Dense layer: Fully linked layers that use ReLU activation to learn high-level information and make predictions. The final dense layer includes the ‘softmax’ activation function typically used for multiclass classification.

Figure 6a–d shows the training of the sequential CNN model. We trained the model twice, 30 epochs each; however, the model was first trained for 18 and 16 epochs on the second attempt due to early stopping implementation. We were able to increase the accuracy from 59–61%. The model trained relatively well; there are still signs of overfitting; however, the validation accuracy curve seemed to follow an increasing trend despite the validation loss not decreasing.

2.8.2 Handgun recognition

Figure 7 gives a brief overview of the key stages in the facial emotion recognition model in the form of a flow chart. The key stages are further discussed below.

Figure 7 outlines a streamlined process for handgun detection and alert generation, composed of three core steps, shown below. This workflow demonstrates a robust mechanism for enhancing safety in real-time environments by integrating AI-based object detection with actionable alert systems. The simplicity of the process allows for quick deployment and reliable operation in high-risk scenarios.

1. Image acquisition: This step involves capturing real-time video or image data using cameras installed in the monitoring environment, such as ridesharing vehicles or surveillance areas. The acquired images serve as the raw input for further analysis, ensuring continuous monitoring to detect potential threats. Live image footage of different handguns in hand was acquired in this study via the laptop webcam.

2. Object detection (handgun): Advanced object detection algorithms, such as YOLOv4, process the captured images to identify handguns. This involves isolating regions of interest within the image, extracting relevant features, and classifying them as either “handgun” or “non-handgun” objects. The system uses deep learning models trained on large datasets of weapon images to achieve high accuracy and real-time performance. According to Korlakunta, YOLOv4 is the fourth iteration of the YOLO algorithm, which introduced a single-stage, end-to-end approach to object detection [32]. Unlike standard two-stage detectors, YOLOv4 processes the entire image in a single pass, making it highly efficient. It delivers cutting-edge precision by combining advanced techniques such as a strong backbone network, a feature pyramid network, and numerous detecting heads [32]. The YOLOv4 used Darknet, an open-source neural network framework, to perform object detection in this study. The Darknet repo is available on the AlexeyAB GitHub account [33]. In addition to the Darknet, YOLOv4 uses a configured file and pre-trained weights, available on the DominykasJurkus GitHub account [34]. My thanks and acknowledgement go out to these users. The pre-trained weights serve as a starting point for training.

3. Save and alert: When a handgun is detected, the system performs two actions: The image or video frame containing the detected handgun is stored in a secure database for record-keeping and potential evidence. Also, an immediate alert is triggered to notify relevant stakeholders, such as the ridesharing company’s security team or law enforcement, ensuring timely intervention.

2.8.2.1 Graphical user interface

This study used Streamlit. Mhadhbi states for GUI, “Streamlit is a free and open-source framework to rapidly build and share beautiful machine learning and data science web apps [35]. It is a Python-based library specifically designed for machine learning engineers.” In this study, two web apps were created, 1 for Handgun detection and another for Emotion detection. Figures 8 and 9 display the home pages of the respective web apps. Within the GUI, the saved weights and models generated from the notebooks used for model training are loaded and integrated.

3.1 Facial emotion recognition

The observed results of the model are displayed in Table 1:

Given the small dataset, it was expected that the MobileNetV2 would not perform well on the testing set despite performing well on the trained dataset. The best all-around performance on the selected metrics was the sequential CNN.

3.2 Handgun recognition using YOLOv4

The following statistics and metrics were taken at a confidence threshold of 0.25. It took a total time of 5 seconds to perform all the detections.

3.3 Comparative analysis

• DeepFace versus MobileNetV2:

  • Across all selected metrics, DeepFace significantly outperforms MobileNetV2.

  • This significant difference indicates that DeepFace is more adept at recognizing emotions in the dataset than MobileNetV2.

• Sequential CNN versus DeepFace:

  • Our sequential CNN outperforms DeepFace across all metrics.

  • This highlights that it not only classifies instances more accurately overall, but sequential CNN demonstrates a higher accuracy in identifying true positives among its positive predictions.

  • It is also worth noting that DeepFace is trained on the larger, imbalanced dataset, whereas sequential CNN is trained on a balanced sample dataset.

• Sequential CNN versus MobileNetV2:

  • Sequential CNN displays a notably better performance across all metrics compared to MobileNetV2.

  • This significant difference in performance demonstrates Sequential CNN’s suitability and effectiveness for emotion recognition on the FER2013 dataset compared to MobileNetV2.

• Classifications

  • With an accuracy of 61%, there will be some misclassifications, as seen in Figures 10 and 11; however, most of these misclassifications also occurred within the other emotions and not just the fear emotion and, therefore, should not hinder the overall performance of the app to significant effect.

Sequential CNN is the most suitable choice for emotion recognition on FER2013 due to its superior performance across all metrics. MobileNetV2’s inferior performance across all metrics suggests it might not be the best choice for this study. DeepFace scored more than 50% across all metrics; however, it could not outperform our sequential CNN, with our sequential CNN model scoring more than 60% across all metrics, and this could be due to the imbalanced FER2013 dataset DeepFace was trained on. Sequential CNN is the clear, better-performing model used to recognize the different emotions in the Streamlit App.

Table 2 shows that the model is better at correctly identifying images that contain handguns (TP = 228) than missing them (FN = 84). Given the number of false negatives, there is room for improvement in recognizing handguns in the images that have not been detected.

The model obtains a good false positive score (FP = 32); however, reducing false positives further could enhance the precision of the model, ensuring that fewer images are incorrectly labeled as containing handguns when they are not.

Table 3 suggests that the overall performance is satisfactory, obtaining an accuracy of 0.80. The balance between precision and recall seems slightly skewed towards precision (0.88 vs. 0.73), which suggests that the model potentially missed some actual handguns.

The screenshots of the model in action seen in Figure 12 show that the model accurately detected the handguns with high accuracy, given different types of handguns, different angles, and different backgrounds.