Feature article from the September 2012 AIRblast
Bracing for Impact
By Erin Polgreen
Erin Polgreen is a self-described 24/7 Ladyjournopreneur, who works at the intersection of audience engagement, news innovation, and entrepreneurship. She is the founder of Symbolia: The Tablet Magazine of Illustrated Journalism, is building the Media Ideation Fellowships, and regularly consults for a variety of foundations and media organizations, including AIR. Here, she explains the mysteries of "measuring impact" for independent producers.
Impact. Audience Engagement. Conversion. Measurement. Evaluation. While difficult to parse, these terms all mark a growing trend among public media makers and funders: the effort to reach more people in more meaningful and effective ways.
A combo one-two punch of audience engagement strategy and a framework for measuring impact can lead to content that does this and, at the same time, helps bring in more financial resources to support the creation of that work.
For the last two decades, funders have come under increased pressure to chart the effectiveness of their grant making, according to Charles Gasper, director of evaluation at Nine Network of Public Media in St. Louis, Missouri, which aims to engage and interact with the community through its TV and online content. This changing tide means that the philanthropy world is calling for more numbers, anecdotal data, and statistics that can tell the story of a project's impact as a part of the grant reporting process.
"There is definitely a kind of pressure for them to demonstrate the impact of what they're funding, and so too places like Nine and other entities are being pressured by funders to demonstrate outcomes," Gasper says.
Defining Impact Measurement
But what, exactly, does impact measurement mean? Simply put, it's the process by which media makers and organizations monitor and document the level of influence their work has in the public sphere. It's about measuring how audiences interact with, share, and act in response to the content they consume. Other terms related to impact measurement that you might have heard before include community engagement, which is the strategy for increasing overall audience participation in content, and reach, which is the combined size of audience when multiple distribution channels are added together. For example, total "reach" might include number of listeners, fans on Facebook, and attendees of in-person events.
At its best, impact measurement allows us to effectively advocate for the role of media in society. At its most challenging, impact measurement is a thorny thicket of tech with a short shelf life, a sea of contradictory stats, a hornet's nest of social media platforms.
"We are in an interesting, fascinating transitional period right now," says Joe Richman, executive producer of Radio Diaries, an independent production house that creates programming for NPR's All Things Considered and also produces podcasts and community events based on its programming. "There are many different ways to distribute stories and measure them, but we don't have all the tools. Right now, we're still in murky waters pretending to know what's happening."
Richman's challenges in measuring the quality of audience engagement — or the actions an audience might take after attending a Radio Diaries event or listening to a segment on All Things Considered — are common for independent producers. When content is syndicated through other networks and media outlets, it becomes difficult to track anything beyond an estimated total audience. A lack of standards — or even basic strategies — for measuring impact create a tough double standard in which a producer must tell the story of his or her works impact without a reliable system for doing so.
So where do I come in?
The current impact measurement landscape is a dizzying mix of patched-together programming that varies from outlet to outlet — and has the potential to shut independent media makers out of the conversation entirely. For example, stations are searching for new strategies to measure degrees of audience participation and interaction in order to improve the quality of their work and inform future programming. But, despite the need, there is no one overarching strategy for measuring engagement from station to station, nor is there a comprehensive guide to successfully managing impact — yet.
A number of new tools are emerging that make charting the impact of a piece of content simple. The New York Times will be putting its fellow from the Knight-Mozilla Open News program to work on this very issue.
New pushes for standardized strategy around impact measurement and a suite of emerging tools that are low-cost and easy to implement are changing the tide. In the next few years, impact measurement will be a much more common term at media organizations around the country. It's more important now than ever before to be well versed in these issues to maintain a competitive advantage in a tight marketplace.
In short, the call to measure a story's impact is one that independent producers would do well to heed. It's also undeniably a part of the media industry's future.
Many look to impact measurement as a holy grail that will transform media, make it more effective, make it easier to find funds. Yet the rate and intensity with which the impact measurement field is evolving is enough to drive an independent producer mad. So, how can bootstrapping independent producers effectively chart the impact of their work? Fear not! We're here to guide you through the choppy waters.
Start at the very beginning.
When setting out to produce a piece, think about measurement from the moment the story is assigned. Don't think about systems of measurement yet. Just set out to answer the following questions:
• What communities will be involved in the production of this story?
• Who should be most concerned with the content of this story?
• What are my goals? For example, if you are producing a piece on high school education, are you trying to call attention to a specific issue? That's certainly the case for Localore project Ed Zed Omega, which is using gaming theory to cultivate conversation around high school dropout rates. Sometimes a goal can even be as simple as "this is something people need to know about."
• Whom can I partner with to build lasting conversations around this topic? Are there community organizations I should be working with as well as larger media outlets? Whom could I work with to create in-person impact?
For more information about thinking about impact from the very beginning, read "Designing for Impact," a short report by Jessica Clark — now working as AIR's Media Strategist — and Barbara Abrash that focuses on measuring the impact of documentary films.
Identify measurable elements.
Now it's time to look at measurement, which often boils down to questions of breadth and depth: how many people can you reach, and how deeply can you engage them around an issue? Once you've set goals, it's important to engage any partners or outlets you might be working with and set expectations for the kinds of audience information they can give you.
If you're working with a station or national affiliate, ask them about measures and whether they have internal processes for assessing impact or engagement. This is also a good time to ID systems or measures of influence. Potential measures include total audience, number of times a podcast is downloaded, use of a hashtag on Twitter, or media hits related to the topic you cover. It can also be effective to measure where audiences are coming from on a global, national, or by zip code level.
When working with partners to syndicate or broadcast your work, getting concrete numbers can be a challenging process, according to Jonathan Miller, executive producer of Food for 9 Billion, a yearlong multimedia collaboration between the Center for Investigative Reporting and Marketplace. Miller tries to think about reach from the very beginning, but there are challenges regarding how accurate the numbers can be. "As we write grant proposals, we ask our broadcast partners for the best numbers they have, and the numbers are really pretty vague. They tend to measure the audience by number of listeners per week. I would love to say we had a piece on last night and it was listened to by 2.8 million people."
But, Miller says, "Numbers [alone] aren't a sufficient measure." True impact measurement is also a question of depth, or how a story can move an audience to action. Does a listener Google the issue discussed in the piece? Does a teacher bring this content into the classroom? "If you've got a group of students having a heated discussion in a classroom in Lancaster, Pennsylvania, that's great! It could change someone's life. But it's probably going to be 20 people." Liking, sharing, or moving a news item on social media can also be a signifier of impact.
According to Miller, it's also difficult to figure out how to quantify grand statements set out in grant proposals — i.e., "inform the electorate." He recommends identifying ways to describe the quality of an audience member's relationship to the story — or to create opportunities to promote and distribute your work in additional audiences. For Food for 9 Billion, Miller built a strong set of relationships in academic environments, which creates additional opportunities for meaningful interaction. "A component of [the] project involves educational materials," Miller says. The Food for 9 Billion team has built interactive, Web-based lesson plans and created an interactive world food timeline with students of Cornell University."
Get a Utility Belt.
This past spring, I co-authored a report for J-Lab that focused on how digital-first news organizations approach audience engagement strategies. In it, we provided a list of popular tools for measuring reach and audience interactions, many of which are low-cost or free. You can see the complete list here and here.
The Integrated Media Association also offers a number of analytics and measurement focused workshops that may be of value to the independent producer. "I know there's a hunger for this in the industry," says IMA Executive Director Jeannie Ericson. "Anything I do with measurement … with analytics in the title is a huge success." IMA's educational offerings include "what's new in Google Analytics" or "how to track audience behaviors online before they donate."
Another free tool for mapping the impact of a story is Sparkwi.se, a visual dashboard from Tomorrow Partners. While the dashboard is still evolving, and is by no means a cure-all, Sparkwi.se is still an interesting step forward.
Sparkwi.se represents a smart strategic mixture of philanthropic and commercial funding for sustainability. It also meets a critical need for both publishers and grant makers. It allows for easy contextual analysis through the aggregation of online usage data and can also help media makers connect their data to larger stories.
However, focusing on one tool as a fix-it-all can be dangerous. It's important to treat impact measurement as a tripod, in which partnerships and goal-setting are other key components to success.
"In some ways I don't think it's that different than it's always been," says Joe Richman. "Fifteen years ago, you were still looking for listener letters, and you were still trying to figure out what the impact of a story was. Back then, it was a little bit of smoke and mirrors. Now we have so many tools. … These tools are useful and interesting, but they're very inexact and can be very misleading. … Along with all these tools for figuring out what the impact is, I think your gut is still the most important for figuring out what's worth doing, or how you're going to reach people."
For example, Radio Diaries reaches an audience of millions by partnering with national forums like All Things Considered — yet they still pursue deeper ways of connecting with their listeners. "We [also] do a podcast and we reach 50,000 people," Richman says. "Why do that? Because there's something wonderful about people who come to you, [that are] part of a community of people that follow your work."
Equip vs. Activate
For many public media makers, impact and engagement are tricky territory. For many media makers, directing an audience to make a political decision or act in a certain way is a serious ethical breach.
"The search for impact in numbers can now affect things," says Richman. "It changes story coverage — the way we measure a story changes the story we tell."
But there is a way to draw lines in the sand, and be clear about what role your work — or the organization you work with — plays in a community.
Nine Network doesn't shy away from impact, but they are clear about their organizational goals. "Nine really is interested in impacting its community," says Gasper. "Our real focus is: Is there evidence that we've had change occur in the community?"
Nine's framework for assessing degrees of impact — without editorial overbearance — could be a good model for independent media makers to adopt. Gasper outlined three levels of audience engagement during our interview:
• Developing awareness.
• Knowledge: Does the audience know something they didn't know before?
• Influencing behavioral change: For example, do more people attend the art museum based on coverage from Nine?
The truth is, no media is without influence. Media informs and shapes the world we live in. For example, "We've put out media that encourages people to be educated," Gasper says, and public media needs to be able to "Measure whether or not the level of education is effective."
So what's next?
Even the smallest project can be made to more powerfully connect with communities by following these simple steps:
• Identify your audience and create goals for reaching them.
• Reach out to potential community partners, be they neighborhood groups, schools, or philanthropic organizations.
• Pinpoint places where you can measure audience response, be it Facebook, Twitter, or the number of times someone downloads a podcast.
• Select a few simple tools for measurement.
• Work with your distribution partner and others to make sure you're maximizing on the resources they may have available to you.
While there is still no does-it-all standard for effectively measuring impact, the strategic questions, tools, and actions outlined here should offer a strong starting point for independent producers.
Polgreen lives in Chicago and loves motorcycle touring. You can follow her on Twitter: @ErinPolgreen
Editor's note: A good, related read — Metrics, metrics everywhere: How do we measure the impact of journalism? — from journalist/researcher Jonathan Stray was published by the Nieman Journalism Lab on August 17, 2012.
AIR welcomes inquiries about republishing this feature article in its entirety or in part. Please contact us at firstname.lastname@example.org.