Facial and head movements have important linguistic roles in American Sign Language (ASL) and other sign languages and can often significantly alter the meaning or interpretation of what is being communicated. Technologies that enable accurate recognition of ASL linguistic markers could be a step toward greater independence and empowerment for the Deaf community. This study involved gathering over 2,000 photographs of five hearing subjects, five Deaf subjects, and five Child of Deaf Adults (CODA) subjects. Each subject produced the six universal emotional facial expressions: sad, happy, surprise, anger, fear, and disgust. In addition, each Deaf and CODA subject produced six different ASL linguistic facial expressions. A representative set of 750 photos was submitted to six different emotional facial expression recognition services, and the results were processed and compared across different facial expressions and subject groups (hearing, Deaf, CODA).

Key observations from these results are presented. First, poor face detection rates are observed for Deaf subjects as compared to hearing and CODA subjects. Second, emotional facial expression recognition appears to be more accurate for Deaf and CODA subjects than for hearing subjects. Third, ASL linguistic markers, which are distinct from emotional expressions, are often misinterpreted as negative emotions by existing technologies. Possible implications of this misinterpretation are discussed, such as the problems that could arise for the Deaf community with increasing surveillance and use of automated facial analysis tools.

Finally, an inclusive approach is suggested for incorporating ASL linguistic markers into existing facial expression recognition tools. Several considerations are given for constructing an unbiased database of the various ASL linguistic markers, including the types of subjects that should be photographed and the importance of including native ASL signers in the photo selection and classification process.