Studying NPR One’s data: “You’re like the first doctor who ever saw an X-ray”

With its NPR One app, NPR has been experimenting with a new way of delivering local and national news to an audience listening on mobile devices. In an interview on a recent episode of our podcast, The Pub, NPR One Managing Director Sara Sarasohn talked with host Adam Ragusea about the app’s purpose, how it works, and what the network is learning about users’ listening habits and preferences. This transcript has been edited for clarity.

Sarasohn: NPR One started out being Pandora for public radio. Because the initial idea is this: In the digital listening space, Pandora just dominates; Pandora creates an expectation of a couple of things and people listen to stuff. They want to be able to skip, and they want it to be smart and personalized for them in some way. We knew that we needed to have what we do — public radio storytelling — fit into those kinds of expectations somehow, and that is the heart of NPR One.

Current: If I were to dial up NPR One right now — say, how about I do it?

Phone: This is NPR One. From NPR News in Washington, I’m Jack Speer. As the Greek government hopes—

Current: What’s happening right now?

Sarasohn: The owner of the phone has turned it on before, and the phone figured out that they’re in Washington, D.C., and that WAMU was the local station, and so [it] has connected you automatically with WAMU. You’re going to get a national newscast; if WAMU has a fresh local newscast, you would get that after that, and then you would get an offer of perhaps a podcast, or you would hear more national news mixed in with more local news from WAMU as well.

Current: And I can skip it any time, so let me go ahead and skip the newscast.

Phone: It’s Tuesday, July 7, 2015, I’m Elliott Francis, WAMU 88.5 News. Just ahead we’ll get the latest on today’s forecast of the metro D.C. region, but first, in local headlines, last week we told you about celebrity chef José Andrés—

Current: Pretty cool. I don’t care about local news in Washington; I live in Georgia, so I’ll skip.

Phone: Hello, NPR One listeners, I’m Stephen Thompson—

—and I’m Linda Holmes. To listen to our latest episode of Pop Culture Happy Hour, hit the “Listen Now” button on your screen.

This week we talk about Magic Mike—

Current: How long until stories go into NPR One?

Sarasohn: You mean from when they’re on the radio? Anywhere from between 20 minutes and an hour, depending on where it is on the clock, what time of day it is, how fast the person cutting them is making the cuts. Pretty quickly.

Current: So there’s no concern about creating a lag to preserve the value of listening to your local radio station?

Sarasohn: Right. We get it up there as quickly as we can. It’s as fast as it goes on the NPR website.

Current: So ultimately it will go on the air first, but pretty quickly after that, because I’ve seen it go up during the show sometimes.

Sarasohn: Yes, they start cutting during the show.

Current: How is it working out? How are your numbers?

Sarasohn: One number that I’m really, really psyched about is our listening time. Our average session length is about 33 minutes now, and that’s been creeping up over the last year since our launch.

Current: How many listeners do you think you have?

Sarasohn: I really can’t say; it’s growing all the time. Another number that we look at a lot is retention, and in the industry a very common way to look at this is, from the number of people who have installed your app, how many are still using it 30 days later. For a news app, the average is about 11 percent of people who install are still using it 30 days later. We’re more than double that. We think we can do better, but we’re more than double that.

Current: That’s phenomenal. What is your goal? What is your eventual vision for this?

Sarasohn: I suspect you’re a lot like me, and that all I’ve wanted to do since I was 8 years old is tell people stories in their ears. And when I was 8 years old, when I was starting out my career, the way that you did that is radio. That’s what we called it, and it was radio. And all I want to do is to keep public radio in people’s ears.

Current: One of the wonderful things about NPR One is that you are able to track a listener’s behavior in ways that we wish we could do on the radio but can’t. You’re able to see how long people listen to something, how long it takes for them to skip it, and you just get phenomenal data. What have you learned?

Sarasohn: We can tell how long people listen to each story, when they skip, how often they skip, if they share it. If you want to give it our equivalent of a thumbs-up, you hit a light bulb if you think it’s interesting.

And when I first started looking at this data, I showed sort of the prototype of that to Robert Smith at Planet Money, and he called me up and said, “Sarah, you’re like the first doctor who ever saw an X-ray.” And I said, “I know, right?” Because beyond any kind of ambition that we have for audience or reach for NPR One, for us, as people who make the craft of radio, getting this kind of information about how people listen to it is brand-new and a little scary, but also priceless.

Current: And it’s been information that we have been able to apply to radio work generally. Can you give me an example?

Sarasohn: One thing to think about is, we’ve been looking at web pages for 20 years, right? And we have not agreed on how to do that. Back in the ’90s we talked about hits, we talked about time spent on page, we talked about pageviews, we talked about unique users over a 30-day period. So if we can’t agree on how to quantify how people look at web pages after 20 years, then we certainly cannot say after a few months of NPR One, “This is how we’re going to look at the data.”

That said, I did a big investigation with Nick DePrey, our innovation accountant. We looked at skip patterns, particularly on intros for radio stories, to see if there were some kinds of techniques that we could say, “Oh, these are things that people tend not to skip.” Or, perhaps, “These are things that people tend to skip.” And I did this in conjunction with Alison MacAdam, who is running our new training unit.

Nick and I went through this big long process, and we identified at the end of it 33 low-skip intros. And after we did that with the data, we did the thing that is totally crucial about looking at the data, which is, I listened to all 33 intros. I sat there with my hands in my lap, and I did not read the transcripts; I just listened to them.

Two things are overwhelmingly common in the low-skip intros. The first is, most of them are 22 seconds or shorter; and the second was, they could have started with a time peg — “Today, this happened.” Instead, they did not do that. They started with a sharp topical statement, so it was very clear what the big idea was. And often later on there would be something like, “Today they filed a lawsuit,” or whatever. But the beginning of the intro was a sharp, clean topical statement.

Current: So no, “Today on Capitol Hill, a cloture motion…” or anything like that.

Sarasohn: No, and most of those intros in that montage, and most of the intros that I listened to, could have started that way. And certainly later on in the intro, there’s like, “The Supreme Court decided this,” or “There’s a bill…” or whatever. And kind of as a check, I listened to a bunch of intros that got very high skips, and most of them were 22 seconds long, so my conclusion was wrong, right? So I listen to the intros, and a couple of things really jumped out at me.

Current: Anybody hear the common thread?

Audience member: Who cares what NPR thinks?

Sarasohn: Exactly! Who cares what NPR thinks? And what I think the lesson there is, is that it might be our value, that we’ve got this series or [we’re] following something up, but maybe that is not the value for the audience. And if you want to convince them that something is important, that they should pay attention to it, that’s not the way to convince them that something’s important. I’m not saying that you must do these things or you must not do these things. What I’m saying is that this is something to think about as you’re writing your intros.

There are other ways that we can handle these. Claudio Sanchez, who’s one of our education correspondent, said, “But don’t you think it’s important that we let the audience know that we have gone down to look at the schools in New Orleans every month this year?” And I said, “Absolutely. But maybe instead of putting that in the intro, you could try saying something like, ‘I’m going to school every month this school year, and every time the teachers tell me the same thing.’” Maybe that’s a more powerful way to communicate that same kind of information, and maybe this will give us permission to break out of this formula that we have where you must say that it’s a series in the intro, and [to] put it in where it has more meaning.

Current: One of the most contentious things that I have heard said about NPR One is the question of, when you’re in a city that has more than one NPR member station, like Boston or Atlanta or Washington, D.C., what will the app default you to and how will it decide?

Sarasohn: I actually think that this is another thing where we have one value and the listeners have a different value. I think that, in the public radio system, figuring out how to make that be as fair as possible is very fraught, and we spend a whole lot of time making sure that you go to this station or that station. And, in fact, a lot of suits spend a lot of time going back and forth about how that whole mechanism works.

But what we hear from a lot of our listeners is, “I live in San Francisco, and we’ve got two stations, KALW and KQED; I want to hear stuff from both of them.” We also hear from a lot of listeners, “I live in Oregon, but I grew up in New Jersey. I want to hear local from both.” So I think the way that we need to think about this is not just how are we going to hold onto our local turf, but how are we going to give people what they consider to be local and not what we consider to be the local that they should be getting.

Current: All beautiful, but the question remains, what does the app do?

Sarasohn: What happens is it geolocates you. There’s a formula that is used both by NPR One and by npr.org, because when you to npr.org it’s got your member station there, too. A lot of people spend a lot of time working out what that formula is who are not me.

Current: Gotcha. So blame the math if you’re mad—

Sarasohn: —or the suits.

Audience member: I’m just wondering if there are plans to take this beyond newscasts into other kinds of programming.

Sarasohn: You mean the local part of it?

Audience member: Absolutely.

Sarasohn: Right now there are places for stations to put in their local stories and their local podcasts. Not all of them do; we are always trying to bring in more. The biggest indicator for people coming back more often to NPR One is when they hear a local newscast. We’re not interested in local because it’s politic; we’re interested in local because that’s what the audience wants, and we really need to build out more all of the tools to get that local in to make sure that people hear it.

Audience member: What does the funding model look like for NPR One, and are there plans for local stations to be able to participate in that as well?

Sarasohn: The funding model is very much actually like it is for the radio right now. We have underwriting credits just like we do on the radio, and there’s also a pledge message. I have to say that the pledge system as it exists on NPR One right now is in its baby stage. But I also think that it is the place that is the most ripe for experimentation and innovation.

We have this idea that you need to pledge during a week or a time period. But really, with NPR One, we could do things like, “You’ve been listening for 100 hours. Isn’t it time to pony up?” We can look at the behavior of listeners who do pledge and try to figure out what made them pledge and then give that same kind of experience to other listeners. There’s tons and tons of stuff that we can do with it.