Application of Artificial Intelligence for Real-Time Facial Asymmetry Analysis

Abstract

Restoring facial symmetry is a major treatment goal of facial paralysis. Surgical techniques have evolved with the aim of reconstructing the paralyzed side as similar as possible to the nonparalyzed side. Therefore, a quantitative measure of facial asymmetry is important for plastic surgery operations. Several graphic systems to quantify facial asymmetry have been proposed. However, they are often time-consuming, may require expensive and complicated hardware, and are not widely used.1 Recent advances in artificial intelligence have opened new avenues for quantitative analysis of facial paralysis.1,2 Recently, Dusseldorp et al. proposed that the “probability of perceived emotion,” calculated by means of artificial intelligence software, can be used as a universal outcome measure for facial reanimation surgery.2 However, this proposal has yet to achieve consensus.3 Although the perceived emotion communicated by the overall face is important, facial symmetry is an essential evaluation criterion for plastic surgeons. Therefore, we have developed real-time facial asymmetry analysis, a type of artificial intelligence designed exclusively to evaluate facial asymmetry. The software has several salient features. First, it enables real-time computation and display of oral and eye asymmetry. Second, a commercially available laptop computer is the only setup required. Moreover, it is available as free software. Real-time facial asymmetry analysis is designed for use with a built-in webcam on a current standard laptop computer. It uses a recently developed artificial intelligence algorithm4 to automatically detect 68 facial landmarks in a real-time captured or prerecorded video. As the major target of surgical intervention is the disfigurement of the mouth and eyes, we have defined facial asymmetry as the displacement ratio of oral commissure and eyebrow vertex (Fig. 1). On completion of analysis, the time-series data of the oral displacement ratio and eyebrow displacement ratio are automatically saved. Users can perform the post hoc analysis with common data processing software (e.g., Excel; Microsoft Corp., Redmond, Wash.) (Fig. 2). Real-time facial asymmetry analysis is available as an EXE file, freely downloadable from our website (https://www.h.u-tokyo.ac.jp/plastic/contents/RFAA.html). It is accessible with a simple click on an icon, and users do not require programming skills to use the software.Fig. 1.: Screenshot of real-time facial asymmetry analysis. Example of application of real-time facial asymmetry analysis on left facial paralysis (analysis of a prerecorded video). Sixty-eight facial landmarks are automatically detected (purple dots). Points A and B denote right and left vertex of the lower two-thirds of the face, respectively. The facial midline (L) is calculated as the perpendicular bisector of the line segment AB. Points C and D are the right (nonparalyzed) and left (paralyzed) oral commissure, respectively. Real-time facial asymmetry analysis computes an “ideal” position of the oral commissure (C′) on the paralyzed side, as a mirror reflection of the nonparalyzed oral commissure (C), in terms of the facial midline (L). The oral displacement is defined as the distance between the “real” and “ideal” oral commissure (i.e., the length of segment C′D). The oral displacement ratio is calculated as C′D/AB × 100 (percent). Points E and F denote the right and left vertex of the eyebrow, respectively. The “ideal” eyebrow vertex position E′ is calculated similarly. The eyebrow displacement ratio is defined as E′F/AB × 100 (percent). This figure is an example of a prerecorded video. For real-time video, the screen projects a mirror image.Fig. 2.: Outcome tracking of a facial reanimation operation. Example of application of real-time facial asymmetry analysis for a case of latissimus dorsi muscle transfer facial reanimation surgery. The preoperative and postoperative oral displacement ratio and eyebrow displacement ratio while vocalizing the sound “ee,” are compared. The surgery was meant to restore oral commissure movement, and the upper face was not treated. The oral displacement ratio was clearly reduced after surgery, whereas the eyebrow displacement ratio remained unchanged.The oral displacement ratio and the eyebrow displacement ratio are simple and intuitive asymmetry measures of the mouth and eyes. These measures could be useful in designing static/dynamic reconstruction and in assessing surgical outcomes. An example of an operation, shown in Figure 2, demonstrates the real-time facial asymmetry analysis tracking of an outcome of a latissimus dorsi free functional muscle transfer. [See Video (online), which shows facial asymmetry analysis of preoperative and postoperative movies. The original video data of Figure 2 are shown. This is a case of complete left facial paralysis caused by external auditory canal carcinoma removal. The postoperative video was recorded after a period of 6 months.] In addition, real-time facial asymmetry analysis has a potential application in treatment of facial paralysis, including mirror biofeedback in facial rehabilitation.5 Visual indicators and real-time feedback of the oral displacement ratio and eyebrow displacement ratio may enable efficient rehabilitation. {"href":"Single Video Player","role":"media-player-id","content-type":"play-in-place","position":"float","orientation":"portrait","label":"Video.","caption":"This video shows facial asymmetry analysis of preoperative and postoperative movies of a facial reanimation surgery (the original movie data of Figure 2). This is a case of complete left facial paralysis due to external auditory canal carcinoma removal. The postoperative movie was recorded after a period of 6 months.","object-id":[{"pub-id-type":"doi","id":""},{"pub-id-type":"other","content-type":"media-stream-id","id":"1_9s3xezyi"},{"pub-id-type":"other","content-type":"media-source","id":"Kaltura"}]} We have used recently developed artificial intelligence technology to develop novel facial asymmetry analysis software. As artificial intelligence advances, it is expected that more useful applications will be invented through a process of trial and error. We are willing to continue improvement to attain the best measures for facial paralysis patients. This study was approved by the institutional review board of the University of Tokyo Hospital. PATIENT CONSENT Patients provided written consent for the use of their images. DISCLOSURE None of the authors has received any financial or other support or has any financial or professional relationships that might pose a competing interest.

Topics & Concepts