You go to war with the data you have

Academic research, at its best, is about asking interesting questions and then designing experiments to answer them. One of the toughest challenges is designing experiments that are both possible and capable of answering the question at hand. The gap between the set of possible experiments and convincing experiments can sometimes be a vast one.

Just like the former US secretary of defense, you go to publication with the data you’ve got, not the data you’d like to have. And different data can lead to some very different conclusions, which helps explain why two competent groups of researchers can answer a question very differently.

I’m interested in questions of whether the internet, and specifically the rise of the read/write web, is broadening or narrowing the perspective of users. On the one hand, the rise of pervasive read/write media means that lots and lots of people are creating content – tens of millions by some counts – which should make a great wealth of perspectives and viewpoints available to the internet user. On the other hand, the structure of digital media means that it’s very easy to select just the media you’re interested in… a scenario Nicholas Negroponte termed “the Daily Me“.

Negroponte thought the Daily Me was an exciting thing. Cass Sunstein thought it was worrisome and wrote a book (twice!) about the reasons it troubled him. An academic debate continues to rage, pitting Sunstein the skeptic against a long list of cyberenthusiasts, including Yochai Benkler and Henry Jenkins.

This is the sort of question social scientists love to explore: thorny, complicated, but ultimately testable. All we’ve got to do is look at some people who spend a lot of time online and others who spend little and see who’s got a broader view of the world. Piece of cake. (The < SARCASM > tag may not be rendering properly in your browser.)

In October 2004, John Horrigan, Paul Resnick and Kelly Garrett released a paper called “The internet and democratic debate” as part of the Pew Internet and American Life project. They didn’t lack for confidence in their conclusions, subtitling the paper, “Wired Americans hear more points of view about candidates and key issues than other citizens. They are not using the internet to screen out ideas with which they disagree.”

The first sentence of the subtitle is experimentally true – that’s what their research demonstrates. The University of Michigan School of Information and Pew IALP conducted a study which surveyed 1500 Americans, some who used the internet to follow the news, some who didn’t. They asked participants whether they’d heard certain political arguments, in favor or against particular political candidates or sensitive issues. Out of a set of of eight arguments (four for a candidate or an issue, four against), broadband internet users had heard 5.5 arguments, while respondents as a whole had heard 5.2 and non-internet users had heard 5.0.

The tougher question is what significance we should attach to this result. The study sees a slightly larger gap in knowledge between people with strong political opinions and the average respondent (5.6 for strong Kerry supporters, 5.7 for strong Bush supporters) – perhaps people with strong political opinions get broadband so they can track politics more closely? Maybe rich people have more leisure time to hear political arguments and more money to buy broadband connectivity? It’s difficult to determine which variables are correlated without doing regression analysis, which attempts to isolate the inpact of each variable.

The Pew report promises that regression analysis has been done and that “Internet use did have an indepedent and positive effect on the number of statements people heard about the candidates.” Unfortunately, it doesn’t include the analysis, and it’s not clear whether internet use is a major or small factor, or whether other factors are better predictors of having heard a large set of arguments.

More difficult for me to swallow is the connection between awareness of political arguments and diversity of viewpoint. I don’t consume a lot of right-leaning US media (to my detrminent – I’d likely be a better informed citizen if I did), and I’d heard all the arguments on the left and right of the issues presented. I’d be willing to guess that I’d heard many of the “right” arguments from “left” media, sometimes in posts that began, “The right is trying to sell argument A – here’s why it’s wrong.” I’m tempted to conclude that Horrigan et al. found a correlation between internet usage and a more-informed citizenry – a correlation that may not be causal, as one can imagine people with a strong interest in being well-informed might seek out broadband internet, or might be wealthier and likelier to have broadband.

But concluding that internet users “are not using the internet to screen out ideas with which they disagree” seems like it’s blurring an important distinction between knowing the other side’s arguments, and having listened to someone make a persuasive case for them. But the distinction quickly points to the difficulty of determining how one would measure a broadened perspective. Is it knowledge of other opinions? A softening of one’s political stances to acommodate other positions? Are we looking for knowledge, for sympathy, or for some sort of change to demonstrate an ideological diversification that comes from online media?

Henry Farrell, Eric Lawrence and John Sides are intrigued by the same question explored by Horrigan and friends. They approach the topic in a paper titled “Self-Segregation or Deliberation? Blog Readership, Participation in American Politics“. The title invokes a related set of political science arguments – Habermas and others have argued that societies depend on a healthy public sphere where issues can be freely deliberated. Sunstein’s recent research on this topic focuses specifically on deliberation, where he makes the argument that deliberation with like-minded fellows can lead to increased political polarization. (My objections to the Pew study might be summarized by saying “knowing the other side’s key arguments isn’t the same thing as good-faith deliberation.”)

Habermas believes that deliberation is an essential ingredient in a healthy society. Sunstein worries that polarized deliberation can make society worse, and fears that the internet enables this sort of polarization. Farrell abd friends point to another complication – citing Diana Mutz, they see evidence that deliberation with people who hold other opinions can lead to increased tolerance but to decreased political participation.

To test whether internet users are more polarized than non-users and how this might affect their participation, Farrell and friends look at a different set of data, the 2006 Cooperative Congressional Election Study. This is a much larger sample set than the Pew data, covering 36,501 participants, but since the researchers didn’t control the questions asked (they were proposed by a large set of researchers) they’re forced to mine the data they have. 5,481 respondents reported reading blogs, and 3,948 listed one or more blog they read. The researchers coded the blogs and discovered that 2,312 respondents reported reading one of 476 political blogs.

Using this data, Farrell and crew classified blog readers as “carnivores” or “omnivores”. Omnivores consumed both left and right leaning blogs, while carnivores consumed only right or left. There are a lot of carnivores – 94%, according to their analysis. Unsurprisingly, folks who read a lot of left-leaning blogs tend to be politically left, and vice versa across the ideological divide. The researchers see a similar pattern in television news consumption – viewers of Fox News tend to be from the right, while viewers of other networks skew left.

This polarization in consumption appears to be correlated to political polarization. The carnivores are not only more polarized than the average citizen, but roughly as polarized as US senators, folks whose political success often depends on their strong, steady party alleigance. (The CCES political opinion data is designed to be comparable to the NOMINATE data collected on senatorial votes.)

The small group of omnivores – 6% of those who report reading political blogs – don’t seem to face the demobilization Mutz warns of. All the blog readers participate at a higher level than the average citizen, other than right-wing carnivores, who participate at roughly the same level as the average respondent.

Farrell and his collaborators conclude with some confidence that blog readers are significantly more politically involved and polarized than the average citizen, a result that might seem to be at odds with the Pew results. After all, Pew’s study concludes that wired Americans “are not using the internet to screen out ideas with which they disagree.” But if we ignore Pew’s interpretation and just focus on their data, the studies are more compatible. Both Pew and Farrell see internet users as better informed than average citizens. Farrell would likely tell you that some of these well-informed internet users are carnivorous blog readers, who are knowledgeable about political arguments through their voracious consumption of ideologically-compatible blogs.

My interest in these experiments has less to do with questions of political polarization and more to do with interest in international news. Are internet readers more inclined to look for information about other countries, since they’ve got such a wealth of information at their fingertips? Or are they more inclined towards information on their home countries, since they can easily choose to avoid international news. Extrapolating from Pew’s data suggests that wired readers might consult more sources and perhaps consume a more diverse diet; Farrell’s research points to a strong homophily effect, which suggests the possibility of geographic cocooning.

Guess I’ll need to design my own experiments using whatever data I can as a proxy to indirectly answer the question… and hope other researchers find other data and other methods to challenge my assumptions.