An attempt by Philadelphia’s WHYY to measure the impact of its news website has its execs asking bigger questions about the best ways to gauge success in public media.
In July 2013, WHYY needed an accurate and effective way to measure the progress of NewsWorks, the station’s digital news venture, launched in 2010. The station talked with CPB, a primary funder of NewsWorks, about integrating an R&D budget for site analytics into the next phase of NewsWorks’s grant.
“At some point during that conversation, we got to talking about Google Analytics and how many phantoms Google Analytics make people chase,”said Chris Satullo, v.p. of news and civic dialogue at WHYY. The popular analytics service provides data that, according to Satullo,“sound really important but [are] really set up for e-commerce” rather than public service.
CPB’s third grant to NewsWorks, in July 2013, set aside $110,000 for the team to build an analytics tool tailored to public media. The resulting effort has mirrored other dashboards under development in public media, including one from the Public Media Company.
The idea at NewsWorks was to build a dashboard integrating analytics that cover every facet of a pubmedia operation, from pageviews to pledge data to Nielsen Audio ratings, and showing how such metrics interrelate. Such a dashboard would make the tool useful for pubmedia execs in all departments.
WHYY’s dashboard aims to provide analytics that would be more useful within public media. Click for a larger image. (Photo: WHYY)
Success in one department often comes at the expense of another department, said Satullo, citing as an example the fact that while donations rise during pledge periods, listening drops. WHYY sought to develop a tool to take such relationships into account.
“More and more, we started thinking about one of the real utilities of the dashboard being to provide executive-level staff at a station with [an] at-a-glance, every-morning look at how the organization was doing on all its key goals,”Satullo said.
Drawing inspiration from the approach to baseball statistics popularized by the book and film Moneyball, Satullo and the NewsWorks team sought “WAR-like metrics”for pubmedia engagement. In baseball, WAR, or wins above replacement, is a number that reprints a player’s overall value to the team based on his role in garnering wins for the team. A public-media equivalent would integrate disparate metrics into a single statistic providing a quick overview of a station or project’s value.
Such an approach wouldn’t yield the most accurate representation of all available pubmedia metrics, but its ease of use would still prove valuable, Satullo said.
“Even for all the flaws of the numbers, you’ve got one combined number you can compare over time,”he said. “If it’s going up, you must be doing something right; if it’s going down, you must be doing something wrong.”
Different tracking methods
Between November and May, WHYY researched and built a clickable proof-of-concept for the dashboard, contracting design work to Philadelphia-based R&D firm Think Brownstone. Satullo presented the prototype June 21 at the annual meeting of Public Radio News Directors Inc. in Arlington, Va., and will also share it at the Public Media Development & Marketing Conference in Denver July 10-12.
The team’s CPB grant ran out June 30, and WHYY is in talks with the funder to receive a follow-up grant.
Meanwhile, Public Media Company is building a measurement tool incorporating much of the same data as WHYY’s dashboard. “The system is deserving of, and in need of . . . a true database,”said Ken Ikeda, PMC’s chief strategist. The company hopes to roll out the tool in 2015 and aims to make it useful for managers throughout public media.
Other efforts are underway to measure impact of public and nonprofit media as well as their performance with listeners and web users. The Center for Investigative Reporting has developed the Offline Outcome Tracker, which its staff uses to catalog the qualitative success of their investigative reports. Metrics used include the staging of community forums, passage of new laws, and the number of legislators who request more information about subjects.
This month, Lindsay Green-Barber, CIR’s media impact analyst, plans to release the results of an in-depth case study focused on CIR’s “Rape in the Fields”reporting series on the sexual abuse of female farmworkers. The project included a 2013 Frontline episode and a 2014 segment on CIR and Public Radio Exchange’s investigative reporting series, Reveal. Green-Barber tracked the impact of the TV documentary to determine how the format’s impact compared to that of other storytelling modes.
News organizations should hunt for new ways to frame their impact, Green-Barber said. “I think the focus should be less on tools and more on methods, because the tools are only as good as what we’re putting into them,” she said. “We don’t know what we want our tools to do, and until we have a really clear idea of what our goals are around impact, then we don’t know what tools we need.”