When Independent Lens unveiled its multi-year content initiative on criminal justice reform in early 2021, its plans included extensive testing of a new technology for measuring the impact of documentary films.
Researchers and filmmakers were to use DocScale, an interactive online tool for collecting real-time audience feedback, for the Stories for Justice initiative, which aims to build dialogue around criminal justice reform.
Stories for Justice was inspired by growing interest in criminal justice issues among documentary filmmakers, audiences, stations and communities, said Lois Vossen, EP of Independent Lens. The Independent Television Service, which presents Independent Lens on PBS, had been developing DocScale over several years, and put the media evaluation tool to the test on behalf of the project.
In addition to collecting survey data on what viewers learned from the films and what actions they would take after watching it, DocScale gave filmmakers and public TV stations insights on how to build engagement around the films.
Early results have been promising. Survey data collected through DocScale’s mobile platform found that the Independent Lens films released during the first year of Stories for Justice were educational to more than half of respondents, according to an impact report that ITVS released in June.
A key goal for ITVS in developing DocScale was to be able to gather useful qualitative data that would help improve the reach, audience engagement and impact of films, according to Grace Anglin, director of research and evaluation at ITVS.
“We really wanted something that would go beyond just a regular survey, or go beyond anything that was already out in the world, to help all these organizations and filmmakers track impact,” said Vossen.
ITVS built and refined DocScale through partnerships with tech developers, funders, filmmakers and public TV stations. The current version, which ITVS began developing in 2020, has been used with 28 films, according to ITVS. It was designed for mobile devices and can work for both virtual and in-person events. Participants at screening events can easily access the survey through QR codes.
The survey is composed of three key modules and has flexible setup options. It collects basic demographic information then asks questions to determine how viewers engaged with a film and what actions viewers would take as a result of watching it, according to Anglin.
DocScale also applies research that enables it to judge how likely people are to follow through on statements that they will take action, Anglin said. In some DocScale surveys for Stories for Justice, respondents were able to follow through within the survey itself. The surveys provided links to websites with information about who in their community has power over issues in question in the film, for example. DocScale can also track how many survey participants follow those links.
DocScale is also designed to build engagement during screenings. As it generates survey answers from participants, it shares those responses with other viewers in a way that provides space for them to react, creating dialogue and engagement around themes within the film, Anglin said.
“A lot of our films look at racial inequality and systemic racism,” Anglin said. “What we were able to find out is that for some people, what they wanted was better tools to have those hard conversations. … That fundamentally shifted some of the work we do because we realized, ‘Oh, what we really need to do is help people be better facilitators of different conversations around this particularly difficult issue.’”
For the first year of Stories for Justice, Independent Lens gathered real-time reactions from participants in virtual and in-person screenings. Among more than 1,700 viewers of Stories for Justice documentaries, 60% of survey respondents said they learned “a lot” or “a great deal” from the films, according to the impact report; 45% indicated that they were “extremely” likely to discuss what they learned with members of their community.
Reactions to some Stories for Justice documentaries were very powerful: 91% of respondents who screened Belly of the Beast, a 2020 Independent Lens documentary directed by Erika Cohn, said the film raised their awareness about the central issue in the film — forced sterilizations of women in California prisons.
Belly of the Beast is one of three Stories for Justice films highlighted in the report. The character-driven narrative chronicles investigations into reproductive and human rights violations in California prisons, including the Center for Investigative Reporting’s 2013 expose of involuntary sterilizations of incarcerated women and prison reform efforts to end the practice. It also documents a campaign to secure reparations for sterilization survivors.
The filmmakers developed an extensive engagement strategy around the film, hosting more than 40 screenings and discussions of the documentary.
Cohn’s goals for the project were to spread awareness of the issue and to support the reparations campaign. She used DocScale to gauge the film’s impact and said it was especially helpful during virtual screenings that had become standard practice for theatrical releases during the pandemic.
“If you’re there for a screening, you can see who the audience is, how they’re responding — you can feel their emotions by being in the same place,” Cohn said. “Because our entire release was virtual, we were able to see … how the film moved the needle in [viewers’] approach to reproductive justice, prison abolition and ultimately getting involved in the film’s campaign, which was crucial to ensuring that reparations passed.”
Most respondents to DocScale surveys indicated that they planned to discuss what they learned from Belly of the Beast, according to the impact report. Many said they planned to get involved in prison-reform efforts or would contact elected officials. DocScale gave participants a concrete way to do that. The platform provided a link to the film’s website, which included a link to an online petition collecting signatures supporting reparations for survivors, Cohn said.
California banned the practice of forced sterilization in 2014, but the campaign for reparations didn’t succeed until after the film’s release and national broadcast premiere on Independent Lens. Last summer, the California legislature committed $4.5 million to compensate sterilization survivors in the state’s 2021-22 budget, according to the Los Angeles Times. The Stories for Justice impact report credits the Belly of the Beast for generating public awareness, advocacy and media coverage that pushed the campaign over the finish line.
“After years of movement-building, and the film creating some political will for this to happen, reparations were passed in California for forced sterilization survivors,” Cohn said.
‘Delving deeper’ into issues
ITVS has restricted access to DocScale to filmmakers who are connected to Independent Lens and a small group of public TV stations that helped develop the tool. That’s because using the platform still requires support from ITVS. But now the indie film funder and distributor plans to expand the circle of public media filmmakers and stations who can use it. ITVS is talking with stakeholders about a wider rollout, according to a publicist representing ITVS.
During various phases of development, DocScale has been funded by the Bill and Melinda Gates Foundation and the Google News Initiative. Stories for Justice was funded by CPB, the Ford Foundation, the John D. and Catherine T. MacArthur Foundation, the Wyncote Foundation and the Park Foundation. (Current also receives funding from the Wyncote Foundation.)
Five stations helped ITVS to develop the tool, three of which, including Oklahoma’s OETA, Louisiana Public Broadcasting, and PBS North Carolina, still use DocScale.
PBS North Carolina has used the platform for virtual events tied to three Independent Lens films, according to Josh Clinard, associate producer. Survey data generated from the events provided insights on which films — and which national issues — resonated most with local audiences, he said. The tool has been especially helpful in refining the state network’s audience-first content strategy and tactics for virtual events, he said.
“When we’re thinking about different programming that we’re doing and … programming that’s able to spark local conversations,” Clinard said, DocScale helps his team “delve deeper” into issues that are affecting North Carolina communities.
“Our recent strategic plan has been really centered around ‘audience-first’ and providing content that our audience is really seeking, and through [DocScale], we’re able to do that,” Clinard said. Through screenings of the film Bedlam “we were able to bring in local community activists and public policy people within the mental illness space in North Carolina to continue to have those conversations.”
Bedlam, a 2020 Independent Lens documentary, examines the national crisis of people suffering from untreated mental illness. Responses from the DocScale survey of participants in the virtual screening prompted PBS North Carolina to host a second virtual event with a panel of public health officials, researchers and local activists who focus on community mental health issues. That event, produced in June 2021, was connected to the PBS debut of the four-part series Mysteries of Mental Illness, produced by Boston’s GBH.
Because of the audience insights PBS North Carolina’s content team learned from DocScale, the state network now routinely hosts panel events with local stakeholders after screenings, according to Clinard.
“For filmmakers, this information is vital,” Vossen said. “It helps them understand if they actually achieve their goal, which is to make a film that will actually educate and inform people,” Vossen said. “… We really feel the more people who use it, the stronger the tool, the better the tool.”
Correction: An earlier version of this article misquoted Grace Anglin’s description of how ITVS intends to use data from DocScale. Its goals are to improve the audience reach, engagement and impact of ITVS films, not to improve the films themselves. The article also incorrectly described how DocScale invited participants in Belly of the Beast screenings to sign a petition demanding reparations for victims of forced sterilizations. The platform did not provide a direct link to the petition. It linked to the film’s website, which included a “Get Involved” section. A link to a change.org petition was among the digital resources included on that webpage.