"You can have it all."

Everything, everywhere, all at once! You could be forgiven for thinking the 2023 Oscar-winning Best Picture title is actually a description of a digital health sales pitch. If you work in healthcare you’ve likely been overwhelmed with click-bait titled emails, slick powerpoint presentations and charismatic sales pitches that promise  “innovative” apps, “ground-breaking” programs and “cutting-edge” devices. But when actually rolled out with high-risk patient populations, the results of these programs are often lackluster.

So how do you pick the winners from flops? The key is to ask targeted, hard-hitting questions that cut through to what's really important. Asking the right question can be the difference between rolling out a program that genuinely moves the needle on patient outcomes and HEDIS measures, and a program that is an expensive failure.

In this article we're going deep into a topic that is familiar but by the end you'll see with new eyes; genuine engagement metrics. You'll finish this article armed with 4 key questions to ask that will allow you to cut through the spin to what's really important.

Why understanding member engagement is so important

Engagement is arguably the most important metric of any member-facing program. After all, if patients don’t engage with the digital health program, how can it affect outcomes?

Unfortunately, there is a lack of consistency in the way the digital health industry defines member engagement. This allows for vendors to spin “engagement” in a way that sounds great for them but ultimately doesn't deliver results.

For example, one company may boast of their ability to “sign-up” 70% of members, but if a majority of those members don't progress beyond submitting their email address it’s unlikely you’ll see much in the way of outcomes. On the flip-side, a digital health company might boast of their fantastic program completion rates, whilst conveniently omitting that they only ever succeed in enrolling less than 2% of their target population.

Member Engagement → Behavior Change → Outcomes

This equation should be tattooed on the forehead of everyone in digital health. We are all trying to improve health and economic outcomes, but that can only be achieved by getting a person to change away from an unhealthy behavior to a healthy one. That might be taking their medications on time, or walking 5,000 steps, or removing trip hazards in the home, or not buying that bucket of fried chicken.

However, behavior change can only be achieved if a member is engaged with the person trying to help them. They have to trust them, they have to listen to them, and they have to do what they recommend. That's why your first priority should be to understand if a vendor's member engagement metrics are genuine.

Let's dig into the 4 questions you should ask a digital health vendor to get to the hard truth of their engagement metrics:

1. How do you define and measure engagement?

2. What is your enrollment rate and how is it calculated?

3. What is your retention rate and how is it calculated?

4. What is your daily engagement rate and how is it calculated?

Bill Lumbergh (Office Space, 1999) ahead of the digital engagement curve.

Q1. HOW DO YOU DEFINE AND MEASURE ENGAGEMENT?

There are two things you want to hear your salesperson say here;

1. That engagement is proactive by the member, and that it's

2. Measured as a member completing a task that is beneficial to their health

'Proactive' means the member is initiating the task or action themselves. Things that don't fall into this category are unanswered coaching calls, messages sent to them, push notifications or a cursory app opening. Be wary also of measures like app-click, number-of-conversations-had or educational articles read. These appear to indicate a patient who is engaged with their own health but can actually just show passive consumption.

Instead what you want to hear is that engagement is the member completing a health task. They did their 15-minute exercise, or attended their medical appointment, or took their controller medication. These are the only things that will ultimately improve outcomes, so all other metrics are just fluff.

OK, your digital health salesperson is sweating a bit but still feeling in control of the meeting. Now you come in with Q2.

Q2. WHAT IS YOUR ENROLLMENT RATE AND HOW IS IT CALCULATED?

The first thing you need to know is benchmarking what 'good' is (although in healthcare it's all pretty bad). Most digital health programs enroll less than 5% of the members eligible for the program. Let's just say that again; less that 5% of those eligible are enrolled. In difficult populations like Medicaid it's even worse; enrolling 2-3% would be considered a great outcome. Unfortunately if you’re missing 97% of members from your program you’re going to struggle to really move the needle on the outcomes metrics that matter.

If the digital health program is promising to blow you away with a much higher enrollment rate, it's worth asking what they are doing differently to get such high enrollment numbers. Also ask how enrollment is done; is it actively undertaken by the member (e.g. do they need to sign-up themselves?) or is the digital health company effectively "enrolling" members based on an eligibility file?

Finally, make sure you understand what constitutes "enrolled." There's a world of difference between 1) a member who created an account online and 2) one who downloads the app, opens it and completes a health task. Make sure the health vendor is explicit about the metrics underpinning their enrollment rate claim.

Your digital health salesperson is really starting to worry about their pitch and tugging at their collar for air. It's time to ask Q3.

Q3. WHAT IS YOUR RETENTION RATE AND HOW IS IT CALCULATED?

Again we need to benchmark industry standards for retention rate. Multiple recent meta-analyses of over 100,000 chronic conditions patients using digital health apps have determined the average patient only sticks with a digital health program for 4-6 days.

Once again that's 4. To. 6. Days.

No one can argue that this is long enough to drive meaningful long-term behavior change or real health outcomes. So by understanding retention you can straight away eliminate the majority of digital health programs that walk in the door. There is no hard rule for what this number should be, so use your common sense based on your member population. Can you get meaningful behavior change in a month? Two?

It’s also important to understand how the vendor is calculating retention, especially if you want to compare with competitors. The two most common are:

1. The average number of days members stick with the program, or

2. The percentage of participants that stick with the program for a certain number of days

The former is "the average user sticks with us for 63 days" and the latter is "our 30-day and 90-day retention rates are 60% and 32%." The benefit of sharing 30-, 90- and 180-day metrics is it gives a picture of member drop-off rates, which shows how good the program is at keeping people engaged once they have joined.

Finally, as we explored in Q1, it’s also worth asking what a patient needs to do in order to be classified as “retained.” Again, is it proactive engagement or receiving unsolicited messages?

We're on the home straight now. Your digital health salesperson is visibly twitching and frantically eyeing the exits. You've got them on the ropes.

Q4. WHAT IS YOUR DAILY ENGAGEMENT RATE AND HOW IS IT CALCULATED?

We’ve discussed the rate at which patients start in the program (enrollment) and how long they stay with the program (retention). But what about their behavior in between?

This is "frequency of engagement" and can be measured in a number of ways. It could be a metric like the average number of sessions per member per day. Or it might be a monthly measure. The digital jargon used here are DAU and MAU. These stand for "daily active users" and "monthly active users." Throw these around in conversation and you'll automatically sound like a pro.

DAU/MAU (the ratio of daily active users to monthly active users) effectively shows how habitually members engage with the solution. In consumer apps a DAU/MAU of over 20% is said to be good, and 50%+ is considered world class (this is where Facebook sits).

However, in digital health even leading behavioral apps struggle here. The top twenty behavioral health apps in the App Store achieve a DAU/MAU of ~15%, which is basically one session per week.

Again, there are no hard-and-fast-rules for what this number should be, as the frequency of use needed for success may vary depending on the program. For example, a monthly 30-minute coaching call might get the same outcomes as an app used 2-3 times a day. But it's fair to conclude that programs requiring habitual self-management are more likely to succeed as the DAU/MAU ratio increases.

Everything Everywhere All.... Oh Forget It

To sum up, you don't need Everything Everywhere All At Once. You just need to know exactly what you're going to get, so you can make a decision about whether it will work for your members.

We encourage you to start by digging into engagement metrics because everything starts there; Member Engagement → Behavior Change → Outcomes. To understand genuine engagement metrics you should ask four overarching questions:

1. How do you define and measure engagement?

2. What is your enrollment rate and how is it calculated?

3. What is your retention rate and how is it calculated?

4. What is your daily engagement rate and how is it calculated?

Answering these will allow you to gain valuable insights into the potential successes and failures of a digital program.

From Engagement to Evidence

However, this is just the start of the journey. Now you are a Engagement Metrics Black Belt, it's time to turn our attention to the digital vendor's evidence base. As with engagement, there is plenty of opportunity for half-truths and fudged metrics when it comes to the evidence that claims are built off.

Be sure to look out for the next article where we help you distinguish between a rock-solid, peer-reviewed evidence base and something Jeremy from Marketing whipped up in his lunch break. We'll equip you with the 4 questions you need to ask to ensure the outcomes you are promised are the outcomes you get.

About the author

Scott Taylor is Co-Founder and CEO of Perx Health, a digital health company changing the way health plans engage with their high-risk members.

Perx enrols a higher proportion of members, interacts with them more frequently and keeps them engaged for longer than any other digital health program. They achieve this by tailoring behavioral motivation strategies to the individual, ensuring the completion of 90% of critical daily care tasks like medication adherence, physical therapy and attending appointments. 

Perx Health has already helped over 30,000 patients achieve better health outcomes and partnered with over a dozen healthcare organizations. Email us hello@perxhealth.com to learn more. We're always happy to chat.