Student: Marwa Saleh
Project: Multimodal Person Identification through the Fusion of Face and Voice Biometrics
Poster: Vertical (PDF) | Horizontal (PDF)
Institution: Lafayette College
Major: Electrical and Computer Engineering
Advisor: Ismail Jouny
Abstract
Person identification is used in a variety of everyday applications from access control and security cameras to social media and face ID on smartphones. The process of identification involves different biometrics such as face, voice, fingerprint, etc. There are still, however, various limitations to their efficiency causing them to be not completely reliable. For instance, in a public setting like that of a museum, using one biometric alone becomes challenging due to factors like background noise, overlap of people’s faces, varying angles and/or distances from the camera, as well as the recently introduced challenge of face masks. This paper focuses on the face and voice biometrics in particular and takes advantage of their fusion to produce a more accurate final decision.
The Michigan State University (MSU-AVIS) dataset is used. It contains 50 subjects whose faces are captured from different angles and voices are recorded as they read from a random script. For the face dataset, videos are first split into frames which are passed into the Viola-Jones algorithm for face detection. Haar-like features are then extracted from each face and used as an input to a convolutional neural network (CNN). For the voice dataset, each audio file is split into samples and any silence/unvoiced speech is removed. Two features are extracted: pitch and Mel Frequency Cepstrum Coefficients (MFCCs) which are then passed into another CNN. Finally, outputs of both CNNs are fused at the decision level by choosing the output of higher confidence in each case.
About Marwa Saleh
Marwa Saleh is a senior Electrical and Computer Engineering major and a Mechanical Engineering minor at Lafayette College. She is a Clare Boothe Luce scholar who worked on joint research between Lafayette College and NC State University. The research aimed to develop an educational game about environmental sustainability which is currently being exhibited in the NC Museum of Natural Sciences. This was an inspiration for her current research about multimodal person identification which serves as a tool for conducting learning analytics for the game. At Lafayette, Marwa takes pleasure in helping other students as a peer tutor at the Academic Resource Hub and a PARDner mentor for first-year students. She is also part of the Lafayette Chapter of IEEE and a member of the Lafayette Leadership Education Committee. In her free time, she loves going outdoors for hiking and rock climbing.