Video anomaly detection aims to identify unusual events within video sequences. Approaches that rely on reconstruction and prediction have shown impressive success in this area and continue to be actively explored. However, recent techniques often rely on pre-trained external components such as optical flow or segmentation maps to handle the challenge of removing background that constitutes a substantial portion of video content. These additional modules, due to their computational complexity, introduce a noteworthy trade-off, particularly in real-time video anomaly detection, where timely monitoring is paramount. So, this paper introduces an efficient video anomaly detection method (MorphVAD) that is based on a novel morphology-based masking module (Morph-Mask). The Morph-Mask harnesses the straightforward concept of morphological transformations to create masks that highlight foreground elements. So, our MorphVAD strategically incorporates these masks during the training phase, focusing on retaining only foreground-related information in memory. Through illustrative experiments and various evaluations, we demonstrate the efficiency and effectiveness of these masks detection, showing the significant enhancements in video anomaly detection performance.