Ubeau

Medium: 

Magic Leap 1 AR Glasses App

Role: 

AR UX Ideator/Designer/Engineer & Researcher

Tools: 

C#, Unity, Magic Leap 1 Lumin SDK, Google Cloud Speech-to-text

For: 

NYU ITP Master's Thesis

Year: 

2022

Collaborator(s): 

Process

{{sect1}}

Overview

The speculative design of a novel user experience paradigm for comprehensive mental health care planning, coordination, and management in cross-platform augmented reality / 3D.

a cross-platform mixed reality / 3d app
a cross-platform mixed reality / 3d app

{{sect2}}

Inspiration

My main research interests as a student at ITP were speculative futures, embodied cognition, and holistic health. My typical outlet for these interests consisted of rapid prototyping various digital therapeutic interventions for mental health, exploring designs for a broad range of conditions from public speaking anxiety to addiction cravings to MDMA harm reduction and ADHD. Reflecting back on these experiments during my thesis research, I began imagining how they could be integrated into a larger whole, sort of a meta-intervention, i.e., a mental health intervention planning tool.

Research Questions:

  • How can new clinical information system interfaces enhance communication within provider-patient and provider-provider relationships?
  • How can richer ways of interacting with health information improve quality of care and user experiences within the current US mental health care system?
  • How can we improve outcomes and expand the resources available for planning and enacting comprehensive, compassionate care?
  • What kinds of tools can help mental health care services become more engaging, participatory, situated, and integrated?

{{sect3}}

Approach

The speculative spatial platform ubeau includes:

  • Magic Leap, smartphone AR, and desktop accessibility
  • cloud-based streaming of encrypted EHR and related data
  • novel interfaces for coordinating plans of care and applying holistic treatment perspectives
  • a speculative library of novel digital therapeutics
  • personalized research and lifestyle-planning tools

{{sect4}}

Outcome

The project aims to ease the process of coordination, engagement, and awareness of the range of alternatives, benefits, and risks of the diverse aspects of mental health care by making care journeys more captivating, concrete, and out-of-the-box-minded.

For the proof-of-concept prototype, I focused on the third bullet point above (under Approach), presenting a novel interface in Magic Leap-based mixed reality for applying holistic treatment perspectives to a prospective patient's plan of care.

{{sect5}}

Demo

Our final task was presenting our research and designs in a 10-minute presentation. You can view the recording of mine below:

Prototype demo:

Prototype screenshots:

biometric hand gesture login
biometric hand gesture login

empty care ecosystem map
empty care ecosystem map

outer health care piece menu
outer health care piece menu

inner health care piece menu
inner health care piece menu

snapping a care piece into the ecosystem map
snapping a care piece into the ecosystem map

an a* (pronounced "a-star") pathfinding algorithm is used to place care insights on the ecosystem map (represented by white dust trails); these inferences are generated from ambient ai health data analyses of care piece relationships along with latest research findings
an a* (pronounced "a-star") pathfinding algorithm is used to place care insights on the ecosystem map (represented by white dust trails);
these inferences are generated from ambient ai health data analyses of care piece relationships along with latest research findings

example of an emergent care insight that can be added to a patient's care plan
example of an emergent care insight that can be added to a patient's care plan

{{sect6}}

User Flow for the Plan of Care Proof of Concept

user flow diagram
user flow diagram

{{sect7}}

Tech Stack

tech stack diagram
tech stack diagram

{{sect8}}

Thesis Requirements

For my master's thesis at NYU ITP, we were tasked with:

  • exploring and beginning to articulate our specific interests
  • developing a level of expertise in an area that we find important or interesting
  • producing a proof-of-concept, body of work, or culminating project
  • presenting the work we did in a coherent and engaging way

{{sect9}}

Research Details

Background Context

My exploration of mental health themes led me to previous NIMH director Thomas Insel's book Healing: Our Path from Mental Illness to Mental Health. In it, he states clearly that the US is going through a mental health crisis, as shown by such devastating statistics as:

  • a 33% increase in suicides in the last 20 years
  • a 20-year average loss in life expectancy for those living with mental illness
  • 60% of people with a mental health condition being without care
  • and only 5% of those in need having recovered

Insel asserts that we have all the treatments we need, yet lack accessible, quality, and comprehensive long-term care supports.

As part of a solution to closing the treatment and quality gaps, he proposes

  • "a coherent, integrated platform that works for patients and providers and, yes, is supported by payers. The platform needs to include objective, continuous measurement (digital phenotyping), a range of digital interventions (crisis response, peer support, coaching, therapy), and improved care management (coordinated care with digital dashboards, quality measures, and integration within the care system)."

I was inspired to begin prototyping some features of such a platform in mixed reality, a medium that I thought could effectively convey the holistic interrelationships of such a range of information sources while providing a greater sense of agency over and enthusiastic engagement with care journeys.

Primary Research

As part of my research, I reached out to various mental health providers to learn more about the on-the-ground reality of Insel's descriptions of mental health care today. Here are some key insights I gathered:

  • some clinics are seeing 4x as many patients now vs. pre-Covid
  • therapists have very little time between sessions, and psychiatrists typically spend only 15 minutes with each patient
  • therapists give referrals to other types of care providers but rarely communicate about their patients beyond the handoff
  • it doesn't always help to have more data, but it would help to have more agency over that data

{{sect10}}

Slides from the Live Presentation

(Recreated with static screenshots from the video since my laptop with the presentation file was stolen soon after!)

Process

{{sect1}}

Overview

The speculative design of a novel user experience paradigm for comprehensive mental health care planning, coordination, and management in cross-platform augmented reality / 3D.

a cross-platform mixed reality / 3d app
a cross-platform mixed reality / 3d app

{{sect2}}

Inspiration

My main research interests as a student at ITP were speculative futures, embodied cognition, and holistic health. My typical outlet for these interests consisted of rapid prototyping various digital therapeutic interventions for mental health, exploring designs for a broad range of conditions from public speaking anxiety to addiction cravings to MDMA harm reduction and ADHD. Reflecting back on these experiments during my thesis research, I began imagining how they could be integrated into a larger whole, sort of a meta-intervention, i.e., a mental health intervention planning tool.

Research Questions:

  • How can new clinical information system interfaces enhance communication within provider-patient and provider-provider relationships?
  • How can richer ways of interacting with health information improve quality of care and user experiences within the current US mental health care system?
  • How can we improve outcomes and expand the resources available for planning and enacting comprehensive, compassionate care?
  • What kinds of tools can help mental health care services become more engaging, participatory, situated, and integrated?

{{sect3}}

Approach

The speculative spatial platform ubeau includes:

  • Magic Leap, smartphone AR, and desktop accessibility
  • cloud-based streaming of encrypted EHR and related data
  • novel interfaces for coordinating plans of care and applying holistic treatment perspectives
  • a speculative library of novel digital therapeutics
  • personalized research and lifestyle-planning tools

{{sect4}}

Outcome

The project aims to ease the process of coordination, engagement, and awareness of the range of alternatives, benefits, and risks of the diverse aspects of mental health care by making care journeys more captivating, concrete, and out-of-the-box-minded.

For the proof-of-concept prototype, I focused on the third bullet point above (under Approach), presenting a novel interface in Magic Leap-based mixed reality for applying holistic treatment perspectives to a prospective patient's plan of care.

{{sect5}}

Demo

Our final task was presenting our research and designs in a 10-minute presentation. You can view the recording of mine below:

Prototype demo:

Prototype screenshots:

biometric hand gesture login
biometric hand gesture login

empty care ecosystem map
empty care ecosystem map

outer health care piece menu
outer health care piece menu

inner health care piece menu
inner health care piece menu

snapping a care piece into the ecosystem map
snapping a care piece into the ecosystem map

an a* (pronounced "a-star") pathfinding algorithm is used to place care insights on the ecosystem map (represented by white dust trails); these inferences are generated from ambient ai health data analyses of care piece relationships along with latest research findings
an a* (pronounced "a-star") pathfinding algorithm is used to place care insights on the ecosystem map (represented by white dust trails);
these inferences are generated from ambient ai health data analyses of care piece relationships along with latest research findings

example of an emergent care insight that can be added to a patient's care plan
example of an emergent care insight that can be added to a patient's care plan

{{sect6}}

User Flow for the Plan of Care Proof of Concept

user flow diagram
user flow diagram

{{sect7}}

Tech Stack

tech stack diagram
tech stack diagram

{{sect8}}

Thesis Requirements

For my master's thesis at NYU ITP, we were tasked with:

  • exploring and beginning to articulate our specific interests
  • developing a level of expertise in an area that we find important or interesting
  • producing a proof-of-concept, body of work, or culminating project
  • presenting the work we did in a coherent and engaging way

{{sect9}}

Research Details

Background Context

My exploration of mental health themes led me to previous NIMH director Thomas Insel's book Healing: Our Path from Mental Illness to Mental Health. In it, he states clearly that the US is going through a mental health crisis, as shown by such devastating statistics as:

  • a 33% increase in suicides in the last 20 years
  • a 20-year average loss in life expectancy for those living with mental illness
  • 60% of people with a mental health condition being without care
  • and only 5% of those in need having recovered

Insel asserts that we have all the treatments we need, yet lack accessible, quality, and comprehensive long-term care supports.

As part of a solution to closing the treatment and quality gaps, he proposes

  • "a coherent, integrated platform that works for patients and providers and, yes, is supported by payers. The platform needs to include objective, continuous measurement (digital phenotyping), a range of digital interventions (crisis response, peer support, coaching, therapy), and improved care management (coordinated care with digital dashboards, quality measures, and integration within the care system)."

I was inspired to begin prototyping some features of such a platform in mixed reality, a medium that I thought could effectively convey the holistic interrelationships of such a range of information sources while providing a greater sense of agency over and enthusiastic engagement with care journeys.

Primary Research

As part of my research, I reached out to various mental health providers to learn more about the on-the-ground reality of Insel's descriptions of mental health care today. Here are some key insights I gathered:

  • some clinics are seeing 4x as many patients now vs. pre-Covid
  • therapists have very little time between sessions, and psychiatrists typically spend only 15 minutes with each patient
  • therapists give referrals to other types of care providers but rarely communicate about their patients beyond the handoff
  • it doesn't always help to have more data, but it would help to have more agency over that data

{{sect10}}

Slides from the Live Presentation

(Recreated with static screenshots from the video since my laptop with the presentation file was stolen soon after!)

Outcome

Other work

Want to create something awesome? Drop me an email.

→ Hi@email.com