The coronavirus disease 2019 (COVID-19) has made it mandatory for people all over the world to wear facial masks to prevent the spread of the virus. The conventional face recognition systems used for security purposes have become ineffective in the current situation since the face mask covers most of the important facial features such as nose, mouth, etc. making it very difficult to recognize the person. We have proposed a system that uses the deep metric learning technique and our own FaceMaskNet-21 deep learning network to produce 128-d encodings that help in the face recognition process from static images, live video streams, as well as, static video files. We achieved a testing accuracy of 88.92% with an execution time of fewer than 10 ms. The ability of the system to perform masked face recognition in real-time makes it suitable to recognize people in CCTV footage in places like malls, banks, ATMs, etc. Due to its fast performance, our system can be used in schools and colleges for attendance, as well as in banks and other high-security zones to grant access to only the authorized ones without asking them to remove the mask. The online version contains supplementary material available at 10.1007/s10489-021-03150-3.