Development of A Personalized Training System for Socio-Emotional Skills of Autistic Children Using Digital Twin

1 in 36 children are diagnosed with autism spectrum disorder. Some key symptoms and supporting neuroscience are: lack of emotion interpretation (amygdala theory), not showing sign of acknowledgement (impaired mirror neuron system), and eyes not focusing on the priority area (relevance detector theory). Existing therapy is not highly personalized and challenging to recognize subtle improvement. Current AI models are not tailored to ASD’s unique facial features (asymmetric, side glancing). My research aims to design and implement a neuroscience-based AI co-pilot training system that tailors to autistic children’s individual preferences (subgroups) and unique facial features using personalized digital twins as training partners. I designed the HarmonyVisage system. The blendshape analyzer offers real time feedback on subtle socio-emotional cues and reciprocal eye engagement, and explains emotions using 13 measurable facial features. Digital twins are used as mirroring practice partners. VGG16 (16 layers CNN) salience map detector automatically guides trainee’s attention. HarmonyVisage is the first AI system built on a unique multimodality dataset with ASD specific 3D tessellation and facial features. 4 phases of experiments were conducted with a total of 46 participants (male: 38, female:8, age:6-13) and 3500+ training sessions. Each participant went through 8-week training. 3 out of the 4 subgroups demonstrated statistically significant improvement with p-values 0.027, 0.001, 0.158, and 0.001 respectively. Test and control group study showed digital twin’s positive engagement with p-value 0.014. HarmonyVisage may be used at home for daily training and in therapy clinic to assist therapist accurate assessment. The proposed solution is highly scalable and accessible. In a world transformed by AI, HarmonyVisage is bridging the gap between perspectives and contributing to an inclusive and harmonic society.