As reporting constantly shifts towards new media outlets and platforms, one essential question for journalists stays the same: How do we measure impact? Public media has a lofty mission — to “create and distribute content that is for, by and about Americans of all diverse backgrounds; and services that foster dialogue between the American people and the stations that serve them.” How do we know when we’re achieving this?
PBS NewsHour has been partnering with social science research institute Knology for 10 years to add value to analytics and audience satisfaction surveys by focusing primarily on, and working with, the communities that are most directly affected by the issues we’re reporting on. When NewsHour decided to produce “Searching for Justice,” a series documenting and raising broader awareness about the barriers that formerly incarcerated people face when reentering society, we knew how important it was to get it right.
A big part of our partnership has been about media impact — defining it, creating it and measuring it. Through our work together, we’ve learned quite a lot about how nonprofit newsrooms can do impact research with the communities that are directly affected by the issues they’re reporting on. We’ve also come up with some strategies for helping journalists capture people’s lived experiences, producing stories that engage and identify with specific groups of news users.
In the interest of sharing these, Knology Communications Lead Elliott Bowen sat down for a conversation with NewsHour Managing Producer Patti Parson, NewsHour Deputy Managing Producer Merrill Schwerin and Knology Head of Media Research Jena Barchas-Lichtenstein to discuss the origins, development and outcomes of “Searching for Justice.”
Elliott: What was the impetus for this series, and what impacts did you set out to make with it?
Patti: We’ve long been interested in criminal justice reform and seeing what’s happening in the criminal legal system. Even before “Searching for Justice,” we’d been telling the stories of the people impacted by that system. This reporting showed that there are lots of stereotypes around people who are incarcerated and their families as well. We wanted to do something that would let the public know these are not “the scary people over there” and really let audiences hear and see who they really are.
Merrill: This was around 2017, when people were really starting to look at the toll that mass incarceration has taken on the country. There was a lot of talk at that time about rethinking the criminal legal system. We were following some of this in our coverage and did a podcast, Broken Justice. We also produced an award-winning piece about Ear Hustle, which was done from within the San Quentin prison. Once we started digging a little further into the system, one thing that really emerged was this revolving door: What happens when people are released, and what challenges do they face? We kept hearing about both of these things, but in terms of public awareness, there was so much that was unknown about the difficulties formerly incarcerated people face once they’re released from the system. So we decided to really double down and look at that.
Jena: There were definitely a lot of gaps in reporting. A lot of the media coverage focused on challenges with reentering the workforce, or voting, or finding housing, or dealing with addiction. Journalists were talking about one piece of the puzzle, but not their intersections, and not the systemic nature of these problems. We also did a survey on public knowledge. Because of coverage around elections, a lot of people knew about voting rights concerns but were much less aware about some other issues, like food insecurity. You can become ineligible for some benefits depending on your experiences with the legal system, and then you don’t get access to things we think of as basic needs. Or consider things like supervised or conditional release programs, which require you to keep a job to not be sent back.
Merrill: Another thing that was getting a lot of attention at the time was the “Ban the Box” movement, which had to do with people needing to check off on job applications whether they’d ever had a felony. But to Jena’s point, these kinds of stories were only scratching the surface. I’ve been a journalist for many years, and I had no idea about all the challenges people face once released from prison — in housing and employment, and elsewhere.
Some of the work of Knology helped us with that, to see what gaps were out there and to consider ways of filling them. In particular, we wanted to really elevate the voices of incarcerated people and their families. We all talk a lot about them, but they don’t often become part of the conversation. So that was one of the reasons behind “Searching for Justice.” We wanted incarcerated people and their families to have some dignity from the series and to be part of a national conversation around policy as the country started rethinking criminal justice reform.
Patti: Advocating for more informed dialogue was one of our key goals. In our work, we never advocate for a specific position on anything, but we do advocate for conversations that can help policymakers and normal people figure out what they want in their communities. With “Searching for Justice,” we wanted to show people different models of reform, their upsides and downsides, and — most importantly — show audiences who these people they’re talking about are.
Elliott: Did you anticipate any difficulties in creating this series and having these kinds of impacts? Going into it, did you have any concerns about making something that made formerly incarcerated people part of the conversation — and in a way that looked at the entire system, not just one part of it? And if so, how did you work through any of these obstacles?
Merrill: One thing we knew from the beginning is that formerly incarcerated people are a sensitive population. Even after reentering society, they’re still part of the criminal legal system, which means you’re often working with people who are traumatized.
Patti: And because of that, we knew that winning formerly incarcerated people’s trust could be tricky. Fortunately, we have producers that excel at finding people and having them talk and building up that trust. One of the last pieces we did was about someone who had been released from prison at the beginning of the COVID pandemic and who was appealing to stay out. He ultimately lost his case, but the day before he had to report back, he got special permission to go to a family picnic. He didn’t want us to tell his family, but he told us and let us come to the picnic and see him tell his family. That’s just one example of the incredible trust we were able to build up.
Merrill: In terms of building up trust, it really helped that “Searching for Justice” was a four-year project. Having all that time really helped us grow and nurture our contacts and resources — which included formerly incarcerated people themselves.
In addition to the story Patti shared, there was also one episode of the series about a man who shot a woman when he was a teenager. She was really young at the time, too. She was in the crossfire.
He was finally released after 20 years, and he said the most important thing was for her to forgive him. And he allowed our producers to be there when he met her. And she forgave him. It was all on camera. That just speaks to really determined journalists who are good at what they do and who built trust by having a long time to cultivate relationships with people.
Along with these journalists, we also benefited from having an evaluation team for the project, who provided real-time feedback on the series as we were making it. That was a big part of what Knology did, so Jena, did you want to talk about that?
Jena: Yes, definitely. One major goal for our evaluation was thinking about whether “Searching for Justice” was for people who had been directly impacted by the system. This gets to what Patti and Merrill said about dignity.
To provide feedback on the series, we recruited a panel of people with four kinds of expertise connected with the criminal legal system. First, lived experience. Some of our panelists had been incarcerated or had family members who had been incarcerated. Our second category was professional experience — that is, people working directly in the criminal legal system, including as wardens, public defenders and other staff roles. Then, third, we had researchers on the panel. This was mostly professors and staff members at colleges and universities doing research on the impacts of the system or policy questions. Last, as our fourth category, we had people who worked in reentry. Not prisons or jails, but those who work with people making the transition back home.
We reached out to a ton of organizations and ended up recruiting about two dozen people from a huge range of backgrounds and positions. And our goal was finding out if the coverage was true to their experiences and if they felt like this was for them and not just about them. And we shared that information back to the reporting team to inform their decisions.
Merrill: That representativeness was really key. From the beginning, we decided that if we were going to have a panel of evaluators, we had better have people on there whose positions and experiences encompassed the full range of relationships to the system.
Elliott: So these individuals were reviewing the series and providing feedback in real-time?
Jena: We asked them to review a couple of pieces at a time, always after they had aired. We would ask both for feedback on the specific pieces and more generalizable advice — for example, “Are there topics you’re not seeing reporting on?” — and we shared that with Patti and Merrill.
The evaluation team thought it was important to approach both the topic and the panelists with a lot of humility. Our mantra throughout was “We don’t know what we don’t know.” People were incredibly generous with the feedback that they gave and especially with the tips they gave for reporters. The feedback was often along the lines of “Here’s a detail that maybe is not telling as balanced a story as you want to tell, and here’s why.” Their comments often came in the form of “big picture” questions. For example, they would say things like “This kind of story needs to include more gender perspectives, because it plays out really differently based on gender.”
And then it was our job at Knology to distill that feedback and share it with Patti and Merrill, who then shared it with the producing team. There were a lot of firewalls between the panelists and the producers. The producers were absolutely not in a position where they directly had to answer to the people this feedback was coming from.
As a researcher, I see this kind of work as valuable because you can both measure impact in an ongoing way and actually increase it by using what you learn. At the same time, we take journalistic independence very seriously — our work stops at distilling and sharing this kind of expert-informed advice, and it’s up to the newsroom whether they take it.
Elliott: Was that a new approach for you all? Had you ever done a reporting project where a panel of experts was watching a series as it aired and then providing feedback along the lines Jena mentioned?
Patti: I think it was a new form. Through the evaluation panel, Jena and the Knology team were giving us an independent assessment of the series, which was valuable in terms of figuring out some best practices around the reporting. There were cases where everyone on the panel said things like, “This word is offensive. Please don’t use it.” For example, “felon” or “ex-con.” A more respectful term we started using was “returning citizen.”
Jena: Independent of us, the reporting team also largely shifted from “criminal justice system” to “criminal legal system.” This matches the usage of many panelists, because “criminal justice system” makes a value judgment that many people think is false. But it’s unambiguously a legal system.
Merrill: I’d agree this was a new kind of input. We always encourage any kind of feedback on our work, and we get input from viewers via email, phone calls, social media and lots of other ways. All of this is super important, and so was getting ongoing feedback from a panel of experts coming from different places in the criminal legal system. This was really key; we felt we were asking the right people about our work, and the panelists provided another checkpoint for us throughout the series.
Elliott: And given the examples Patti and Jena just mentioned, it sounds like having the panel provide this independent review really paid off.
Patti: It definitely paid off, particularly in terms of helping us cover all the different aspects of the reentry process and showing how much people’s experiences can differ based on factors such as their backgrounds and where they live. The series gave us an opportunity to talk about things like coming out of prison when you’re gay, or when you had been an addict when you went in, or about how difficult it can be to get an ID. Most people don’t know those things, and covering them allowed us to expand public awareness around the difficulties of reentry.
Jena: That was definitely the panel’s conclusion. What we heard throughout the evaluation was that the reporters did a fantastic job of bringing in different perspectives, and with nuance, sensitivity and respect. The panelists thought they did an amazing job highlighting experiences and stories that often get passed over in favor of the official institutional perspectives on the criminal legal system.
Patti: That’s a real tribute to NewsHour, that we have so many people able to engage at that kind of high level. And I think that really helped in terms of realizing our vision for the series. We knew one obstacle we’d face with “Searching for Justice” would be the fight against breaking news. Finding production time and airtime are always big challenges for projects like this.
Merrill: Agreed. Since we’re a news show and only have one hour every night, getting commitment to something like this can be really tough. And you can understand why. People who come to us every night want to figure out what’s going on in Ukraine, what’s going on with the debt ceiling, and you don’t want to interrupt that process of really being there for the viewers on the big important news of the day. But we’ve got a really robust newsroom doing great work, and that helped ensure that these pieces would see the light of day.
Patti: Because of the strength of the “Searching for Justice” pieces, people knew that these were going to be “getting you” kinds of stories. And that really helped.
Elliott: Now that the series has ended, are there other things this whole project taught you about impacts, in terms of defining, creating or measuring them?
Jena: I can see how this partnership offers a new way to think about impact journalism. What we see in a lot of instances, at least in terms of Knology’s work, is that funders want evaluation to come only at the end of a grant. Frequently, they wait until after a project’s finished to ask “Did we do what we set out to do?” Personally, I feel really strongly that it’s more useful for media organizations to get real-time, ongoing feedback, because that allows you to have more impact as things proceed. If you get periodic updates where people are saying “Here’s what’s working and what’s not working,” you have information you can actually use, in a more tangible way. It’s called formative research, and it allows you to test your theory of impact and make iterative improvements. This is definitely something that happened with “Searching for Justice.” Many panelists commented on how the reporting got better over time, because the reporters had more nuance and more depth of understanding.
Merrill: I’d agree that getting evaluative feedback along the way was really valuable. Having these checkpoints where we could ask “How are we doing?” was really important. It’s not like we were telling the newsroom “You need to do this,” but still, tracking and measuring impact as a process was really important.
Patti: I think we did hit on a new model for measuring impact. Funders are always saying “Tell me about impact,” but they don’t always know if they want numbers, like how many people tuned in or gave feedback in some way, or something that’s more qualitative, like individual responses from viewers. So having a system where we’re consistently tracking the responses of a specific group of experts over a long period of time was really helpful. It helped ensure that what we were producing was accurate, representative, relevant, relatable and respectful. It helped us figure out how to determine what constitutes a “good” impact, and how to best approach it.
Dr. Elliott Bowen is Knology’s communications and writing lead. A proponent of actionable and accessible research, he has contributed to Knology’s media projects through the creation of digital content that communicates key findings to journalists and general audiences in a variety of different formats — including blog posts, social media threads, audiovisual presentations and the Knology newsletter.
Patti Parson became managing producer of PBS NewsHour after serving as a field producer for the program. She started her television career on the crew at KCTS in Seattle, then switched direction to become the station’s director of promotions and later returned to production as senior producer for its public affairs programming. She also helped produce a PBS special and Bill Moyers’ Journal.
Merrill Schwerin is deputy managing producer of PBS NewsHour. Schwerin joined the development team after working for decades as a national and international field producer for NewsHour.
Dr. Jena Barchas-Lichtenstein has directed Knology’s media research since 2017. A linguistic anthropologist by training, they believe that media should respond to people’s informational needs — and they partner with news organizations and other media creators who share that goal. Their collaborations include an open-access guide to reporting on numbers (with PBS NewsHour) and community-based polling for civic science reporting (with WNYC and the Bodega and Small Business Group).