Psychiatry Mentally ill avatars
Medical students practice examination conversations in virtual space.
The diagnosis of mental disorders depends not only on the answers patients provide in the interview with a healthcare professional. The way they behave is likewise relevant: how does their voice sound? Do they look the other person in the eye? What’s their posture like? Are they pacing restlessly? Is the face stony or animated? “An experienced healthcare professional monitors and assesses all these aspects while talking to a patient, noticing changes to the facial expression within milliseconds,” outlines Associate Professor Paraskevi Mavrogiorgou from the RUB clinic for psychiatry, psychotherapy and preventive medicine.
Medical students have to learn how to do that, which isn’t always easy. “At the university clinic, we mainly see patients who are seriously ill,” says the physician. “It is difficult to motivate them to take part in training sessions with the students. Plus, they are not the kind of patients whom the future doctors will most likely be seeing in the course of their career.”
In order to enable students to practise with patients presenting the full range of symptoms and degrees of severity and to broaden the panorama, the clinic has launched a project focusing on creating avatars that feature typical symptoms of different mental disorders as well as psychiatric and psychotherapeutic treatment situations. Students wearing a virtual reality headset (VR headset) will be able to meet those avatars in a three-dimensional space – or, as experts call it, “exploration”.
This has never been done anywhere yet
The clinic has already established rooms and prepared them structurally. Here, small cubicles will be set up that students will enter with their VR headsets in order to talk with their virtual patients. “As far as we know, this has never been done anywhere yet; we are pioneering the field,” says Professor Dr. Georg Juckel, Medical Director at the clinic.
The greatest difficulty: designing a dialogue flow
The preparations have taken two to three years; now, the virtual characters are being fleshed out. “They need a biography, a case history,” explains Paraskevi Mavrogiorgou. “And they must be able to answer specific key questions in accordance with their respective mental disorder.” If the attending doctor asks an avatar suffering from schizophrenia whether he hears voices, the avatar should say he does. If the attending doctor asks a question that doesn’t help find the correct diagnosis, the avatar mustn’t simply remain silent; rather, he has to give a plausible, evasive answer.
“At the moment, this is the greatest difficulty: designing a dialogue flow,” says Georg Juckel. “We have compiled hundreds of pages with instructions and dialogues,” points out Paraskevi Mavrogiorgou. For the dialogue to work, high-performance speech recognition software is necessary to ensure that the computer in the background understands the questions and can provide relevant answers.
It’s the same as with Walt Disney’s animated movies.
Georg Juckel
In order to create avatars with plausible physical features, the medical professionals intend to collaborate with a commercial enterprise specialised in visualisation for medical applications – after Arthur Berns, young computer engineer from Göttingen, has designed the logical framework. The avatars’ body language is to be adapted to the type of mental disorders they represent. They might be sitting, standing, lying or pacing during the interview. Their facial expressions, too, are supposed to tell the trained observer something about the underlying disorder. “The programmer has to design each of these facial expressions individually,” as Georg Juckel elaborates the immense complexity of the task. “It’s the same as with Walt Disney’s animated movies, or with computer games today.”
The avatars are to be built up segmentally, “or else the whole thing would soon become quite boring for the students,” says Juckel. Thanks to the modular system, the difficulty level of the exploration can be regulated. The selected avatar represents a specific psychiatric diagnosis or treatment situation. However, the way students get there, the questions they ask, is not predefined: this is what they will be learning under guidance and supervision.
First steps in spring 2020
The project team expects that the avatars will be taking their first steps in spring 2020. In summer semester 2020, students might begin conducting the first interviews with them. “We’re hoping that this approach will catch on with the students - especially in view of today’s leisure activities,” concludes Paraskevi Mavrogiorgou. They are definitely curious to see what the virtual patients are all about.