Engaged. Amy Bucher
and your client well.
As you write down what long-term success looks like, be as specific as possible. Don’t just write “blood pressure”; write “reductions in blood pressure” or “percentage of people whose blood pressure changes from being high to normal.” The more specific you can be, the better able you’ll be to get the right data to tell your story.
TIP LESS CAN BE MORE
Especially for newer products, a simple outcomes story may be more powerful than a kitchen-sink-style one. Amazon once sold only books; Google used to be just a search engine. Their compelling early successes gave them a platform to expand their purviews. Just like a minimum viable product focuses on the most essential features, your first outcomes logic map should focus on only a handful of truly critical outcomes.
Identify the Behavior Changes Needed
Your long-term outcomes don’t happen by magic. They happen by behavior change on the part of your users. In most cases, this behavior change needs to be sustained over time. To take the example of lowering blood pressure, in order to see measurable differences on tests, people might need to eat differently, start walking regularly, and take their medication every day for weeks, months, or longer. It’s not a one-shot deal.
Focus the metrics in your outcomes map on behaviors that your product can reasonably be expected to influence. If your product is a medication management app for people with high blood pressure, your outcomes plan should focus a lot more on medication-related behaviors than the exercise and diet pieces. You might want to look at those behaviors, too, just to explore whether your users are also making those changes—sometimes people on a behavior change kick will have a positive spillover effect to other behaviors—but don’t hang your hat on them since it’s not what your product is designed to do.
You’ll want to make a judgment call about how granularly to measure behaviors. For example, if taking medication is an important part of achieving outcomes, you could just list “take medication as prescribed” as the behavior of interest. But if you know your users probably don’t take medication yet, you might also include “make an appointment with a doctor” and “fill prescription at pharmacy” as additional steps. That will remind you to coach users through those steps within the product and help you identify where the pitfalls are if people don’t actually end up taking the medication.
It’s very likely that your product can’t directly measure the behavior changes that lead to outcomes. That’s fine! It’s very typical, in fact, since most behavior change happens out in the world and not online inside a digital product. You’ll find other ways to measure whether people are doing these behaviors. What’s important is identifying the behaviors that need to happen for your success story to come true, so you can design your product to affect them.
Determine How to Measure Exposure
The term “exposure” here is a fancy way to ask if people are actually using your intervention. There’s a famous quote from the former Surgeon General C. Everett Koop that I often use when presenting about this topic to healthcare audiences: “Drugs don’t work in patients who don’t take them.” The modern corollary is “Interventions don’t work in people who don’t use them.”
You’ll want to measure usage for at least three reasons. First, theoretically, your product won’t do anything if people aren’t using it. Second, you need to be able to show that people used your product as part of their behavior change process to tell a compelling story about its success. If you put a product out in the world, and a year from now your long-term success metrics have come true, but the people showing those changes never used your product, you’ll have a hard time convincing anyone that you had anything to do with the changes. Third, usage is a “leading indicator”; you can measure it almost immediately after putting your product out into the world. Leading indicators are your earliest evidence of whether a product is successful or not.
Some of the common measurements you might include as leading indicators include:
• App downloads or installations
• User accounts created
• Logins or sessions started
• Actions taken within the program (e.g., articles read, videos watched, action steps checked off a list)
• Return visits
NOTE DON’T STOP HERE
A common pitfall product teams make is focusing too much on leading metrics.2 There are all sorts of reasons why this happens; they’re relatively easy to measure, and they enable quick reports back to leadership if they want to track how a product is doing. While sometimes people struggle to understand the complexity of the more meaningful behavior change outcomes, most people quickly get the importance of the number of users or the frequency of use. It’s okay to use your leading metrics as a success indicator, especially early in your product’s life, but don’t lose sight of the more critical lagging metrics. They tell a much more compelling story.
Some of these metrics may be familiar to you from another metrics planning tool, the conversion funnel (Figure 2.2). If your team uses a conversion funnel to track marketing and acquisition, you can incorporate it into a larger outcomes logic map. Your funnel won’t be so funnel-shaped anymore, but it will do the job of helping you track the right metrics across the product lifecycle.
DIAGRAM BY AIDAN HUDSON-LAPORE.
FIGURE 2.2 A conversion funnel is a type of metrics planning tool that helps teams gauge how well they are reaching, acquiring, and retaining product users.
Keep in mind, too, that more does not necessarily mean better with exposure metrics. Most behavior change interventions need a certain amount of time to work, and more is overkill. If you know what the right “dose” of your intervention is for your users to accomplish their goals, make that the target and not a session more. And if you don’t know, make an educated guess, do some testing, and iterate over time.
Fill in the Specific Data Needed
The last step in completing the outcomes logic map is to fill in the specific data you’ll need to collect to determine if you’ve achieved each outcome. “Specific” is key here: “blood pressure readings” is not as good as “self-reported blood pressure readings using a kiosk at the pharmacy.”
In the process of listing out the data you need, you’ll be able to identify information you can collect through your product itself and the information that you’ll need to get outside of your product. Anything that happens inside the product needs to be accounted for in the design requirements; for example, if you need to have users answer certain questions every two weeks, you’ll need to make sure those questions are included in the product, along with a scheduled prompt so that people answer them at the right times. Anything that happens outside of the product will need to be accounted for in other ways. I’ll cover some of those in the section on “Evaluating for Effectiveness” later in this chapter.
Get a Baseline
Outcomes stories are really stories about change. There’s “in the beginning,” the bad old days, and then the happy ending of today’s outcomes. In order to tell the change story well, you need to know what “in the beginning” looks like. That’s why, once you know what metrics you’ll be collecting for outcomes, you should measure them at the outset to establish your baseline.
If you’ve ever used an app or a program that starts with a long questionnaire, one of the things it’s doing is creating this baseline about its users. If you have information about your specific users