6th Sense

{{sect1}}

Overview

I collaborated with a team of designers, technologists, and researchers at the XR Brain Jam on a mixed reality simulation of schizophrenia symptoms. The project aims to educate first responders, mental health professionals, and family members on some of the most challenging aspects of schizophrenia.

‍

Featured in Unity's Metaverse Minute: 7 ways mixed reality is reshaping training and education

‍

{{sect2}}

Inspiration

Understanding psychosis is critical to effective allyship and interfacing safely with affected persons.

‍

How can AR be used to help improve people's understanding of the condition and increase their respect for those who face it on a daily basis?

‍

{{sect3}}

Approach

We aimed to simulate schizophrenia symptoms in an authentic and respectful way. Our design was informed by our research into academic descriptions and firsthand accounts of schizophrenia, as well as a secondhand account provided by one of our group members who was closely related to someone diagnosed with schizophrenia.

‍

Tools we incorporated:

• We worked with Magic Leap 1 to design spatial audio (simulate hearing voices) and anchor digital objects in the surroundings (simulate symptom triggers)
• We incorporated Google Dialogflow v2 and an avatar made out of particle systems to guide the participant's conversation with a simulated "reality monitor" (typically an affected person's own inner character or a role played by a family member or caregiver)
• As a next step, we would like to incorporate a heart rate monitor that, when detecting a sharp increase in heart rate, triggers the reality monitor to check in on the participant

‍

{{sect4}}

Outcome

In our mixed reality experience, the participant takes a walk in the shoes of someone with schizophrenia under the guidance of a benevolent friend. Typical encounters of daily life trigger auditory hallucinations. The participant turns to their friend to help find their way through.

‍

Our prototype includes the three following trigger scenarios:

‍

1. Paranoid thoughts about a fire alarm

• Reality check: Alarm is real, but non-threatening. Affirms that the participant is safe.

2. Self-deprecating thoughts upon seeing a self-help book

• Reality check: Asserts that tools are healthy, juxtaposing voices. Reassures and encourages the participant.

3. Subliminal messages projected onto a TV news show

• Reality check: Video is real, but the targeted messages are not personal. Contextualizes the clip and comforts the participant.

‍

{{sect5}}

Demo

AI Companion (Reality Monitor) Demo:

‍

Presentation Video:

‍

{{sect6}}

User Flow

‍

‍

{{sect7}}

Tech Stack

‍

‍

{{sect8}}

Slides from XR Brain Jam Pitch