How do news organizations measure impact?
That’s the question I found myself talking with Phil Bronstein of the San Francisco Chronicle earlier this week. He’d gotten in touch to talk about what tools are available to help newspaper editors track audience and reach for their stories, hoping that I’d have some insights on “cutting edge” techniques to track the reach and impact of news stories posted online. We talked a bit about the challenges of social media tracking after the demise of Technorati, the possible benefits of bit.ly-type analytics, questions of influence and reach raised by Klout and similar systems. All well and good, but measuring how many people read a story is something any web administrator should be able to do. Audience doesn’t necessarily equal impact.
There may have been a day in the rosy past of newspapers when a wall between the publisher and the editor meant that newsrooms published only what was most newsworthy and civically important, without consideration of a given story’s appeal to their audience. In an age where editors can know instantly whether a story on a school council meeting is playing better than a story about a labor action, it’s hard to believe that access to analytics doesn’t shape coverage decisions. Some outlets, like the Huffington Post, have embraced this new world to the point where they are poster children for analytics-driven coverage, using feedback from Google Analytics to inform most if not all decisions about story placement and emphasis. This willingness to respond rapidly to market feedback has likely helped HuffPo’s rapid audience and market growth – whether or not AOL’s acquisition of the site was a wise move, most newspaper publishers would welcome ten-figure interest in their properties.
The danger of traffic-based analytics driving journalism is that you may end up with newspapers that look more like Demand Media-style content farms and less like the civic guardians we want and need them to be. It’s certainly fair to observe that newspapers have been audience driven, at least in part, since inception and that some of the shortcomings of contemporary papers, as well as local newscasts, derive from a focus on driving readership and viewership. But adding an analytics into the newsroom puts the question “Is this story reaching a broad audience?” front and center in a way that’s hard to ignore or avoid.
If an ideal editor is making decisions based on what’s newsworthy, and a realistic editor is civic and audience concerns, how do editors determine whether they’re successfully serving both masters? What are appropriate analytics for civic impact?
As Phil and I talked, I found myself thinking about the LA Times’s coverage of obscenely high government salaries in the city of Bell, CA. In depth, investigative reporting by Ruben Vives and Jeff Gottlieb led to fraud trials, a turnover of the city government, and ultimately to a Pulitzer for the pair of reporters. The reporting on Bell, CA suggests two ways newspapers might measure civic impact: the arrest of bad guys, and the praise of one’s peers and professional societies. But these aren’t exactly quick metrics, and not every worthwhile piece of civic journalism has this magnitude of impact.
Traffic doesn’t seem to be the right measure of civic impact. A story that gets lots of page views or is widely shared might be civically relevant, but might also be salacious – amusing and popular as much of the Anthony Weiner coverage has been, I’m not sure it’s been a positive contributor to our civic involvement. Phil suggested that comments aren’t an adequate metric either. Stories that garner long comment threads could suggest broad involvement, but also may suggest partisan controversy.
I mentioned an idea that I’ve been trying to pitch for a while: in an age of participatory media, news demands participation. Or to quote Benjamin Barber, “People are apathetic because they are powerless, not powerless because they are apathetic.” For people to pay attention to an important story, it’s possible that we need to work to make it possible for people to have an impact on the outcome of the story.
Ideally, we can find better ways to do this than turning our Twitter icons green in solidarity with Iranian activists. Reporting on local civic issues offers the possibility of connecting people to opportunities for action in their own communities. And if newspaper web sites start trying to broker these connections, we gain another possible metric – the efficacy of a story in connecting people to community organizations, volunteering opportunities, and other forms of civic engagement.
That’s not a comfortable role for newspapers to take, Phil reminds me – it smacks of advocacy journalism. But the Bell, CA story is another form of activist journalism: by relentlessly shining a light on political malfeasance, Vives and Gottlieb were demanding that someone take action against these corrupt officials. Eventually, both citizens and prosecutors did. The difference between what I’m proposing and what the Pulitzer winning reporters did is that I’m suggesting newspapers link to possible solutions and measure how effective at driving engagement they are.
This would be far from a perfect metric. It wouldn’t tell you how many people read a story on homelessness, and then sought out community organizations on their own to volunteer with… though adding a feedback cycle where local organizations could communicate changes in community involvement to newspapers might. And it wouldn’t track one of the most critical functions of investigate journalism: the fear it generates in politicians and corporate actors that they could end up on the front page of a newspaper if they break the law. Clay Shirky is worried that losing this deterrence effect is one of the dangers of losing “accountability journalism” in the transition from broadcast, offline models of journalism to participatory, digital ones.
My point is not that I’ve got good metrics for civic engagement for newspaper journalism… or any journalism. It’s that we need to be thinking about finding and developing them. What we measure, we become. If we measure only how many people view, like or tweet, but not how many people learn more, act or engage, we run the risk of serving only the market and forsaking our civic responsibilities, whether we’re editing a newspaper or writing a blog.
Hi–this is something I’ve been really interested in (first as a reporter and now as a student), so I’d love to know more about ideas for how to measure this. This could be done through a social audit-type approach… few news organizations do social audits now, and most audits don’t cover this kind of thing.
Would love to hear more about social audits, Kate – how would you suggest a paper go about doing that? Probably isn’t going to address my hope for near-realtime metrics, but sounds like a really worthwhile addition to what newspapers should be doing to measure impact.
Nice vet of the challenge, Ethan. I was really hoping you had solved this dilemma! In public broadcasting, where social return on investment (SROI) has been eluding us for decades, we continue to satisfy ourselves that audience = public service. (And the more loyal, the more they support the effort.) But a) this borrows from the commercial ratings marketplace, b) it doesn’t give us that better target of journalistic impact to aim for, and c) we can get big ratings without any local news effort (just play NPR national programming) which begs the question of why local stations should invest in local news (or exist in the future)? I recently made a list of some 15 quantifiable outcomes of our work… none resembling the better bulls-eye… but maybe we need *a host of metrics* to get at something as difficult as SROI.
Building on the Guardian’s social audit model is one option–they address editorial but could do so much more to evaluate impact (still, the fact that they address editorial at all seems to put them miles ahead of most news organizations).
I would like to see newsrooms adopt a strategic planning process–set goals that align with their missions, formulate concrete plans to help them achieve those goals, and periodically evaluate their progress. If they don’t take the time to figure out how to measure impact, they’ll just fall back to what’s easy to measure–like traffic.
Hi, Ethan:
I’ve been pursuing these questions for a few years now with different intellectual partners—Tracy Van Slyke with the impact summits we did last spring; a set of documentary filmmakers and funders to think about how to assess impact of longer term projects, and with Knight, which released an impact assessment guide for community info projects earlier this year. There are many juicy questions still to be answered, which I’d love to talk to you about more if you’re interested.
Ethan, for the sake of completing the logical inference, is there evidence you’d cite proving that newspapers under the editor/publisher-wall model were of a consistently better quality? How was success measured in the 20th century? In circulation? Revenues? Pulitzers?
Pingback: Joho the Blog » Berkman Buzz
Jessica, what I’d really like is for you to come to Civic at some point this fall and talk with all of us about what you’re doing and how we might work together…
Pingback: links for 2011-07-02 | Beyond the Echo Chamber
Isn’t that what oneline polls are for?
From another perspective, news outlets have been trying since the 1980s to pander to what they PERCEIVE to be audience tastes, with the result that news has become filled with crime, scandal, and fluff. However, I would argue that, despite many focus groups and ratings services, editors and journalists still usually only guessed at what they thought would interest their most prominent demographic. I have great optimism that most people, even (or especially) young people, want real, good news (i.e. local politics, relevant culture), and that perhaps these new metrics will lead to a turn away from what has come to stand in for good journalism.
Ehtan,
Which groups organize,host, and moderate in person community meetings to discuss civic issues raised by media? Newspapers could tie-in to such meetings. Radio and tv talk shows allow some listener contributions but depend mostly on experts or story authors. Perhaps newspapers can put links to organizations on their websites that try to help address the problems that appear in the articles. I may start a related business soon. Keep me posted.
Pingback: Hvordan mÃ¥le journalistikkens gennemslagskraft? « Ny Journalistik
Pingback: Tensegrities » Blog Archive » How do we measure impact?
Pingback: Hudin » The future of journalism for now
Pingback: » Quantifying impact: A better metric for measuring journalism The Linchpen
Pingback: Papo On - Diário do Nordeste
Pingback: The billion-dollar question: What is journalism for? — Tech News and Analysis
Pingback: The billion-dollar question: What is journalism for? | Electronic Staff
Pingback: Nuove metriche per misurare l’ impatto del giornalismo | LSDI
Pingback: Measuring the impact of journalism « intelligent measurement
Pingback: What Journalists Can Learn from Apple’s Map Mishap « Groundswell
Pingback: Come misurare l’ impatto del giornalismo? | LSDI
Pingback: Which News Organizations Influence Wikipedia? | Tow Center for Digital Journalism
Pingback: {learning} The Art of Evaluation_#ICM820 | Goals Go Global
Comments are closed.