BRAVO - preventing falls

Years ago, when I was studying design at university in Amsterdam, I had plenty of opportunities to try out methodologies and design approaches on smaller assignments. I decided to keep one of these school case studies online, as I think it creates an excellent contrast to how I felt about the design process back then. It is interesting to compare the school work where we could go all out with ideas to real-world commercial projects, which you can read in some other case study on this website.

01
Introduction

In the Netherlands, every 5 minutes, an older person is admitted to the hospital due to a fall accident. Fortunately, there are ways to reduce someone’s risk of falling. Fall prevention training or a medical examination are just two of them. However, the elderly living on their own often underestimates the risk of falling. The goal of our class assignment was to:

Prevent elderly living on their own from unexpected falling
02
Digital life technology

The university paired every year for these semestral projects with some company or laboratory. This year’s partnership was with the Digital Life Laboratory, which pioneered an experimental technology of monitoring people in their households. Based on the Kinect motion capture technology and new algorithms, they could produce data about the movement and physical status of an individual. It could predict the risk of falling based on changes in postures. The technology got a name: BRAVO. The laboratory wanted us to focus on making use of the technology in our class assignment of preventing the elderly from falling.

An illustrative image of the technology from the Digital Life centre website.
03
28 UX researchers

Everyone was working a couple of weeks on explorative research. The class separated into groups that focused on three main areas:

  • professional caretakers (home caretakers, general practitioners),
  • unprofessional caretakers (family members),
  • elderly - target group (empathy research).

The research has been extensive, and I will mention a couple of highlights:

In the empathy research focus, we have built a suit that could simulate the physical burdens of people above 60 years old. Anyone could put this empathy suit on and harden the movement, walking, vision, hearing. We have been doing a couple of tests with it. People tried to use a phone or carry a plate with a glass. Some people had enough by just walking a couple of hundreds of meters.

On the left: The separate parts of the empathy suit
On the right: Putting the suit on one of our classmates.

One of the research groups prepared a culture probe kit. It is a kit of things which included: a disposable camera, a city map with some dot stickers and a notebook. Participants of the research captured their daily lives with these tools.

The culture probe kit delivered to participants.

We did dozens of interviews and fly on the wall studies during the span of weeks. Also, before starting the project as well as during the iterations. We built two personas and empathy maps of our target group.

04
The key moment

Since 28 people were working on the same project, each of us needed to find a more specific area of focus. Together with my professor, I decided to take on a crucial part of the experience. The part when the elderly is already in imminent danger of falling. My challenge was:

How to persuade the elderly to take action before they fall? Convey  the message when the elderly are in imminent risk of falling.
05
Touch devices

As I nudged a little beforehand, it was quite clear that if I built something with touch devices, it would be rather complicated. Using the empathy suit, we had a tough time writing a text message or do basic operations on a small screen with touch. Further, the interviews with family members revealed that there was frustration when there was a need to charge portable devices. The elderly often misplace them or forget to charge them, which results in family members not being able to reach them.

06
Interest into new technologies & TV as the favourite

From interviews with the target group, it was surprising that the elderly in the Netherlands are open to new technology that can help them. The real problem remained that often the gadgets are not designed inclusive enough. Onboarding and usage were too challenging for them.
The culture probes & fly on the wall observations showcased that the elderly have life figured out. Apparent daily routines are part of most of the people. When I compared the habits of multiple participants, almost all of them include interaction with a TV device. Which also is part of the majority of the households. TV & remote due to being commonplace in their lives got me thinking that this could be the most appropriate channel to communicate through.

07
Loneliness & stubbornness

The elderly like to talk to people. Usually, living on their own indoors, they tend to feel very lonely. Often seeking someone to talk.
We asked in the interview about long-distance communication through mobile devices. Families revealed that it is rather hard to ask the grandparents to do something through a mobile device. It is either that they don’t feel the personal connection, or they immediately forget what the relative was asking for if they are not in the room with them. When reaching them through communication devices, it is quite hard to convince the elderly to do something. Some relatives called their grandparents stubborn for these reasons.

08
Fall prevention training

I studied official papers from the World Health Organization focused on elderly falling prevention. There are so many exercises to train the body and keep it active. The issue is that everything has to be precise and under the surveillance of a professional. It is easy to worsen the current condition by doing those exercises in the wrong way. Wanting to empower the elderly, this would most certainly result in the opposite.
Ultimately, after speaking with professionals, they were very skeptical of some specific steps. They also revealed, these problems happen when the elderly are exhausted or do much physical activity. The best way to prevent a near fall is simply to just rest for a while. This insight became the critical purpose of my solution. When the device detects an immediate high risk of falling, it persuades the elderly to rest for a while.

09
Fitting into the routine & making it personal

Very early in the process during initial brainstorming, I focused on the emotional part of the experience. After all, it is a delicate situation when there is an immediate risk of falling, and you want to be as nurturing as possible. I chose to use voice as a primary channel to connect with the target group. Also, it is easy to hear a voice in the house. You don’t have to have any device with a screen on you at a particular moment.
Voice is a channel that is familiar to this generation and might help with the persuasion to take action. From previous chapters, it is clear that it might, to some extent, substitute the personal connection the people are seeking.
Knowing the routines from our culture probes research, watching TV is a lot of the times a fascinating part of the days. It is also available in most of the households.
Going into more specific problems, I arrived on a challenge, that:

In the detection of a risk of fall, the product (voice) would guide the person to sit down and relax.
10
The solution

It is pretty straightforward. The device monitors movement in the houses through cameras and evaluates the risk of falling. The moment the algorithm calculates the risk as dangerous, it tries to persuade the user to sit down and relax. Initially, this is done by starting a conversation and explaining that it is time to watch some TV. The device automatically turns on the favourite channel. The goal is not to tell the user about any danger directly but rather keep a positive approach. It would not help to scare and stress people off. If the risk looks out of ordinary and severe, it also calls a professional caretaker automatically on the background. Ultimately the goal is to sit down the user.

The goal is to have the device handle most of the cognitive processing by making the experience as straightforward as possible for the user.

11
Second act by the TV

After sitting down to a TV, the device tries to provide feedback on the current state of the risk. The TV interface encourages by displaying feedback and various types of notifications to ensure the user follows the “relax” routine.

Designing for a TV is different from designing for smaller devices. Usually, a user is watching it from a distance, in a vastly different context. The most important part was the typography. To maintain good readability for the elderly user from a distance, I decided to go with a wider font-weight and large type size. Darkening the live TV content behind the message allowed me to create sufficient contrast and make the user shift their focus on a different element on the screen.

Creating different variations of the notification led me to conduct a small validation session on a TV screen. The goal was to maintain good readability from standard couch distance while covering as little screen space as possible. For displaying feedback on how much time until the relaxing session completes, I wanted to use a circular bar. However, since the TV UI is all about covering as little screen space as possible, I decided to go with a straight progress bar that covers about half the area. Keeping the user motivated to finish the task is the primary goal of the UI. It is done by displaying the progress and mixed motivational messages, still maintaining a positive approach.

An exploration for the TV UI

Side note: Later, after the testing sessions, I found out that TV programs in the Netherlands often contain subtitles. I decided to use a filled white background to show the initial message to not collide with the subtitle

12
Designing a voice AI

During the voice design process, I aimed for two different aspects of the solution. Voice input and audio input. The goal was to create an interface that doesn’t merely have good copy, but also maintains a good conversation. It is crucial to make the conversation feel as natural as possible by creating a good flow between the inputs and outputs. When the device leads a good conversation, its persuasion capabilities increase thanks to being perceived as more human. I knew that to achieve this would need a lot of tweaking and iteration during the prototyping & testing phase.

13
Dummy AI. Voice/TV prototype and the story of a small company in Amsterdam

Arriving at a clear idea about the whole experience and completing the first version of a few visual elements for the TV UI, I went straight into thinking about what would be the best way to showcase this solution and to test it with real users. The journey from an automatically triggered voice interface to displaying data during live TV in one user flow proved to be a rather interesting prototyping case that I needed to plan carefully.

During my university years, I have been experimenting with a variety of tools that could make designs more alive. Luckily I stumbled across a small company in Amsterdam called Framer. I have been participating in their meetups and gatherings for some time. Back then, the tool was a lot about writing code in a simplified version of JavaScript called CoffeeScript. The great thing about a code-based prototyping tool was that you could build anything you wanted, quite fast with immediate feedback, and connect it to any device. Although not being a profound coder, I decided to challenge myself and use the Framer tool to make my sketches & ideas alive.

A few hundreds of lines of code & many headaches with debugging, I managed to create a basic prototype that could recognize human voice by using a connection to Google’s natural language processing API. It could also speak with a British accent by using a link to the built-in Apple text to speech function. Furthermore, it turned on simulation of a TV and displayed notifications combined with different dynamic visual elements. It enabled me to test the solution with real users without having to sacrifice some parts of the experience. A crucial part of my prototyping plan was to do it in a way that allowed me to change what the device is saying with a single command. This enabled iteration over different conversation types during the testing smooths & easy.

Changing & tweaking responses of the system were quickly done by adding a command during testing.
14
Surprising people

The first session took place with a moderator and two potential users from the target group. In the beginning, I wanted to have as little influence on the users as possible. I briefly told them about the device and what it is doing, leaving out the details. The first testing was a profound learning experience for me, as it allowed me to identify multiple previously unseen pain points.
One of the biggest surprises for me was the way people acted at the beginning of the experience. They immediately got surprised by the speaking device and were confused about the proper way to respond. It wasn’t until this point that I realised how uncommon it is for people to interact with voice interfaces. Using voice devices, the user usually triggers the system themselves and starts the interaction (Hey Siri, OK Google, ...). In the case of Bravo, it is precisely the opposite. Bravo starts the interaction with the user, and they are not used to this kind of trigger. This insight significantly altered my thinking about the product. I found out that, first, the user needs a heads-up before the start of the experience, so that they can prepare themselves for the conversation. Second, the device should also ask if it is appropriate to interrupt at that particular moment.

Solution: When the device is triggered to speak (because of a high falling risk), it gives the user some kind of heads up. For example, a dull sound might do the trick. It needs to be long enough for the user to process what sound it is, and also calm enough to not scare anyone. Introducing itself, the surprise of the initial voice message decreased. Also, the device’s first question should be if it is possible to interrupt at the exact moment. If not, the device could try in a few minutes.

15
Two additional tweaks after testing

The whole conversation between the device and the user is too fast and too short. The AI was not able to persuade the user to take appropriate action in such a short timespan.

Solution: Add at least one more reply to the whole conversation flow. Furthermore, it increases the delays between responses.

The quick switch from voice to TV was also unexpected by the user. It was a little scary to see their TV turn on automatically when they were speaking to a device.

Solution: Turn on the TV right when the device starts talking. Turning on the TV simultaneously with the start of the conversation is more intuitive. This sync provides a visual clue at the beginning of the flow that a process is starting. With the TV on, the feedback can be displayed for the user immediately, so that they know whether the device understood them correctly.

This video illustrates a simple product use case
16
The pulsing dot

I am intentionally writing about the details of the interface and the micro-interactions only after describing the testing session. The reason being that usability testing had a significant influence on these factors. The interactions are designed around a single objective - supporting the user throughout the experience. The system displays whether it is the device or the user talking at any point in time to make the interactions smooth. This was achieved by creating a pulsing dot animation and changing its color to blue (device speaking) and green (user speaking). As an affirmation the system writes the AI outputs & user inputs on the screen as a secondary interface.

To further incentives the user to go through with the entire experience, feedback is shown at the edge of the TV screen, displaying the time left until the end of the recommended relaxation. To motivate the user finish the relaxation session.

17
Further development

I can’t help myself, but at some points, I felt like I am making the elderly health even worse by making them relax (possibly wait for the caretaker) while sitting and watching the TV. So after digging more into health research, I found out that there are some easy exercises that the users could do while watching the TV, and their health could benefit from it in the long term. I see potential in making the user flow longer, and by implementing an exercise session after the relax session would help them stay active.

18
The conclusion

From the beginning, this project quickly shifted towards a new unknown territory containing voice-driven interfaces, TV UI, and a new approach to how people and technology interact with each other. I would not consider this by any means as a final product but more of a start exploration into what could be the direction of health devices for the elderly.