“Impact” is a feel-good media buzzword of the moment, increasingly required by the funders of many projects and invoked by some PTV stations, news organizations and documentary producers as key to demonstrating the social good derived from their work.
But defining the concept and then measuring whether a media project has demonstrated its value remain elusive challenges for many. During “Understanding Impact,” a two-day symposium convened last month at American University in Washington, D.C., participants explored a number of the ad hoc systems for tracking impact that are taking form.
Organizations including the Center for Investigative Reporting in Emeryville, Calif., and KETC, the Nine Network of St. Louis, have developed their own methodologies and hired staff members to measure the impact of their work. Others, including Public Media Co. (formerly Public Radio Capital) and the nascent Media Impact Project at the University of Southern California, look to tap into big data sets that are still being assembled; their aim is to follow the broad societal reach of public media institutions and discrete projects that use media to focus attention on a particular issue or promote social change.
The Media Impact Project’s large data set at University of Southern California has the potential to be “a resource unlike any we’ve ever had before,” said Joaquin Alvarado, chief strategy officer at the Center for Investigative Reporting. (Photo: Karen Everhart)
Still others complain about being overwhelmed by data, particularly online metrics, and worry that all the extra work required to collect the data doesn’t produce enough meaningful insights to justify the effort. And some journalists are leery of crossing over into advocacy by being asked to focus on what changes their reporting can elicit.
In public media, the increasing emphasis on tracking impact is directly related to changes in public broadcasting’s business model, said Mark Fuerst, director of the Public Media Futures Forum, which sponsored the April 17-18 event with CIR.
The number of individuals who contribute to public television stations has declined to the point where public radio has achieved near parity with TV in revenues garnered from membership, Fuerst said. Grants from philanthropic organizations are helping to fill the gap.
But these monies come with some strings attached: media funders increasingly want to see analyses showing that the projects they invested in helped to spark a public dialogue or promoted awareness or even change in the way that people think about a societal or public policy issue. As Fuerst noted, CPB is also eager to show Congress that its tax-based funding of public broadcasting is an effective medium for informing and educating the public.
Though the methodologies for tracking public media impact varied widely among the speakers, they shared the same basic objective — to observe measurable change in the public dialogue, community life and public awareness.
CIR, for example, looks for “change in the status quo as a result of a direct intervention, be it a text article, live event, or radio or TV program,” said Lindsay Green-Barber, CIR’s media impact analyst. The center’s methodology also categorizes change by degrees: It can be macro (a political change), meso (an informed public) or micro (a change in an individual’s attitude or knowledge.)
USC’s Media Impact Project looks for changes in “knowledge, attitudes and behavior.” For the Nine Network impact consists of “positive changes in our community that result from our collective efforts,” said Amy Shaw, s.v.p. of community engagement.
Others factor in scale, the number of people affected and their demographics. For example, PRI assesses whether listeners contribute content through online engagement projects, such as its recent effort to collect global lullabies, said Kathy Merritt, v.p, content strategy and development.
Coverage that’s ‘relevant anywhere’
At CIR, impact is woven into new projects from their conception, said Robert Rosenthal, CIR’s executive director, who described it as key to financial sustainability. CIR’s funders “want results,” he said. The organization has been a leader in figuring out methodologies for increasing media impact, and has sponsored other seminars exploring the topic.
The traditional metric for assessing impact would have been whether a story got an official thrown out of office, but the newer goal is to “see how the stories you do change people’s thinking,” he said. The release of a story is only the beginning of the process.
To raise the impact of a project, CIR identifies audiences with a specific interest in a topic. For “Rape in the Fields,” its 2013 film collaboration with Frontline and Univision, CIR convened a post-broadcast meeting with advocates, law enforcement representatives and victims to discuss possible solutions to the problem of sexual abuse of migrant farmworkers. Academics at UC Davis later began research projects on the topic, one measure of the impact the story had far beyond its broadcast.
CIR’s data on the backlog in processing military veterans’ disability benefits was shared with local media outlets, and resulting news coverage prompted Congress and the Obama administration to jump on the issue. Within eight months of CIR’s first reporting on the problem, the backlog and waiting times for benefits dropped markedly. (Graphic: The Center for Investigative Reporting; Sources: Department of Veterans Affairs and CIR analysis)
CIR has also spread the impact of its stories locally by publishing data that can be incorporated into original local coverage. In August 2012 it posted an interactive data map on the backlog in processing disability benefits at the 58 Veterans Affairs regional offices across the country, and in May 2013 it launched an API (application programming interface) for the backlog-related data it had collected, open for anyone to use, with credit to CIR. When local reporters used the data in their own coverage, their stories made the topic “immediately relevant anywhere,” said Cole Goins, CIR distribution and engagement manager.
“The problem with the war is that it seems so remote to people. This brought it home,” said Aaron Glantz, CIR’s reporter on the story, of the local reporting. In the months since CIR began publishing the data, Congress and the administration have jumped on the issue, and both the backlog and waiting times for benefits have dropped markedly.
For Reveal, the public radio series that CIR and Public Radio Exchange began piloting last fall, CIR supplemented its investigative radio feature on opiate addiction among U.S. military veterans by publishing an interactive data map of VA medical facilities with the highest prescription rates for opiates. CIR also created a “how to localize this story” guide, which showed local reporters what information to look for and different angles, said Goins. (Congressional hearings followed and opiate prescriptions dropped.) Built into the online application was the capacity to measure which features were most used, said Green-Barber.
Among the common themes that emerged from the conference was a consensus that any strategy to measure impact needs to be defined when a media outlet begins the project, a practice that CIR follows, said participant Tom Thomas, co–chief executive of Station Resource Group, which provides analyses to public radio audience measurement and service strategies to client stations. He also sees partnerships as a key component to actually achieving results.
“It’s not an accident that some projects have an impact and some don’t,” said Caty Borum Chattoo, executive in residence at AU’s School of Communication.
Shaw
Avoiding advocacy
For the Nine Network, which has made “community impact” an integral part of its sustainability strategy, projects designed to bring about positive community change begin with community partners already on board. “We establish consensus around the critical outcomes needed to make an impact in the community,” said Shaw.
The importance of a thoughtful advance strategy became apparent to PRI following a weeklong five-part series on cancer in the developing world. For the 2010 project, PRI pulled together a large community of influential organizations and individuals. But after the broadcasts, that community quickly moved on, Merritt said, calling it a “missed opportunity for local connections and relevance.” So a new project is working with more local station partners and a longer time frame for stories to roll out, she said.
Merritt also acknowledged the difficulty of getting journalists to buy into conversations about what they want their reporting to accomplish. “We have to find the sweet spot that’s not advocacy — because I don’t think we want to go there — but it’s something that can still be a powerful change,” she said.
While CIR and the Nine Network have already integrated the concept of impact into their work, many organizations are still not operationally staffed to undertake such efforts, said Todd Cunningham, a former Viacom executive who is leading USC’s Media Impact Project. One of the project’s first tasks, a review of organizations that are “culturally ready” for impact measurement, found a major personnel gap that will need to be addressed.
Defining public media’s metrics
For those organizations that are ready, big data sets are being assembled that could potentially help.
Public Media Co. is assembling a database geared specifically to impact measurement for public radio and television. It will track “audience, financial and engagement metrics across all platforms and activities,” including how much money stations spend per person within their broadcast area to produce local content.
The goal, said Ken Ikeda, PMC’s chief strategist, is “to help stations be intentional about the change that they want to see.” Apart from audience ratings, much of the existing data is based on ad hoc needs of specific local stations, often outdated “and not terribly relevant,” said Erik Langner, managing director. More up-to-date data could help stations improve their decision-making as they seek to articulate their progress to funders and decide where to invest resources.
PMC expects to charge for the data, which will be accessible on automated dashboards, and make it possible for stations to compare their performance to similar stations.
The database was piloted at New Hampshire Public Radio using 25 different metrics. Currently, PMC is working with Oregon Public Broadcasting, American Public Media, Philadelphia’s WXPN, Los Angeles’s KUSC and Colorado Public Radio to determine what data they want and how the public media system at large could best use it.
Other projects in the works are less immediately public media–focused but could eventually be helpful to public broadcasters and producers. The Harmony Institute, a New York–based research center “dedicated to studying the impact of entertainment,” and in particular how stories can create social change, is finishing up a web application to be released by year’s end that maps the media impact of films dealing with social issues.
Dubbed ImpactSpace, the tool will categorize hundreds of films by 40 issues, from education to the environment, with broad categories such as health documentaries broken down into the subcategories of food, nutrition, illness and treatment. Media-makers who use the system can “find the gap” in coverage of issues or see where audiences have already been super-served on a topic, said Clint Beharry, Harmony Institute’s senior creative technologist.
Films on similar topics, such as AIDS, and with similar audience reach can be compared by the types of impact they had, and the system tracks which audiences are talking about each film — those already active in a particular cause or newcomers.
The USC-based Media Impact Project is also building a large data set as it positions itself as “a global hub for collecting, developing and distributing the smartest approaches” to achieving impact, said Cunningham. Producers, distributors and researchers will be able to contribute their insights, with safeguards “so the data is not shown to anyone else who shouldn’t see it,” he said.
While public media is not a specific focus of the project, “we need to challenge USC” to add it to the mix, said Joaquin Alvarado, CIR’s chief strategy officer, calling it “a resource unlike any we’ve ever had before.”
The Media Impact Project is also working with Participant Media, a company founded by philanthropist Jeff Skoll to focus on feature film, television, publishing and digital content that inspires social change, to develop The Participant Index, a “collection of media performance data” for narrative and documentary film, television and online video not limited to Participant Media’s own content.
Chattoo, the project’s lead strategist, said the index, now in its pilot stage, is gathering data from a massive online public opinion survey and will attempt to capture individuals’ attitudes, actions and perceptions after they watch specific film titles. It will also try to assess the “framing” element that a piece of media can bring to a social issue, she said, explaining that even those who haven’t seen, say, the SeaWorld documentary Blackfish may know all about it and the questions it raises about keeping killer whales in captivity.
Burdensome big data
Not all conference participants have bought into the idea of gathering large data sets to measure impact. Chicago’s WBEZ stopped some tracking it had been doing, said Wendy Turner, WBEZ’s v.p. of systems. Reams of data — from Nielsen ratings to social media shares and page views — were generated “but there weren’t any insights being drawn,” she said, and the “weight of the data collection itself started to build some resentment” among the staff who worked to collect the data but never saw any follow-up reports on what it might mean.
“Large databases become their own burden,” said Cory Allen, director of research at Boston’s WGBH. For him, the key to integrating impact into a project is “having key people in the organization who care about it” and a way to share information on what works best.
Some journalists remain wary of the concept of impact for fear it will bleed into advocacy, even at a time when numerous journalism startups — most notably the First Look Media network of digital sites started last year with $250 million in funding from eBay founder Pierre Omidyar —“are abandoning dispassionate neutrality and seeking intentional impact,” said Jan Schaffer, executive director of AU’s J-Lab. Public media, she said, needs to determine where it will fit “in this evolving space.”
In the broader public media world, “not a lot of stations are paying attention” to the concept of impact, Fuerst said. But for those that do, there can be a financial upside. St. Louis’s Nine Network has secured more multiyear funding commitments from foundations as a result of its impact-centered strategy, and philanthropies are giving more money than before, Shaw said.
“My belief is, had we not made the shift, we would be half our size and nothing but a PBS pass-through,” she said.
The Philadelphia-based Wyncote Foundation sponsored “Understanding Impact” through the Public Media Futures Forum directed by Mark Fuerst. Current is funded in part by a grant from the foundation, and managed by American University as a journalism center within its School of Communication, which hosted the event.