SRPP: When Words Speak Just as Loudly as Actions: Virtual Agent Based Remote Health Assessment Integrating What Patients Say with What They Do

This talk is about a multimodal dialog platform for remote assessment and monitoring of neurological and mental health. This platform, guided by Tina, a virtual guide, enables participants to engage in immersive interactions where objective speech, facial, linguistic, and cognitive biomarkers are computed in near real-time from audiovisual recordings. Using the Patient Report of Problems (PROP), Tina encourages patients to describe their most bothersome issues and factors affecting them, capturing their responses in their own words. Combining these subjective patient reports with objective biomarkers offers significant potential to improve telemedicine quality and enhance siteless trials and digital therapeutic interventions.