Home » Blog » Berkman » Evaluating censorship circumvention tools and “irresponsibility”

Evaluating censorship circumvention tools and “irresponsibility”

In a recent blog post, my friend and colleague Evgeny Morozov questions the responsibilities of academics who study internet censorship circumvention tools. As one of the academics Morozov mentions by name, I felt compelled to address his concerns. I should make clear that my response is on my own behalf, not on behalf of any of my colleagues at Berkman or elsewere.

Evgeny’s concern in his recent post appears to be that I haven’t publicly critiqued Haystack, a proposed censorship circumvention tool that’s received a great deal of laudatory press coverage. That’s true. Neither have I said anything positive about the tool on my blog or in the press.

Not all dialog takes place in blog posts or in op-eds In the security field, dialog ranges from in-person, off the record conversations to published scholarly research, and everywhere in between.

It would be a mistake to conclude that the “internet intellectuals” Evgeny calls out in this piece were silent on concerns about Haystack simply because we’ve been speaking privately, not publicly. I’ve offered counsel to several funders of circumvention tools about Haystack, offering concerns that the code and protocols were unpublished, unverified and untested. To the best of my knowledge, none of the people I’d spoken to ended up offering funding for the project. I spoke to any journalist who asked me about the project and offered a similar answer. In a post discussing his involvement with covering Haystack and potential conflicts of interest, Cyrus Farivar makes clear that I’ve expressed a great deal of skepticism to him offline about the project.

I’ve not published on Haystack for a very simple reason: I haven’t been able to conduct a proper evaluation of either the tool or the protocols behind it. I’ve been in contact with Austin for quite some time, seeking access to the code or an in-depth discussion of the protocol. I have high hopes that he’ll allow a version of the tool to be evaluated in the evaluation of circumvention tools we’re scheduled to carry out at Berkman later this year. I have also been in dialog with Iranian activists, none of whom reported using the tool, or working with anyone who used the tool – while this is reassuring that activists weren’t using untested tools, it’s been frustrating, as it’s also made it impossible for us to test the tool in the wild by working with someone using it. (Heap now reports that the tool was tested with roughly a dozen users in Iran, which helps explain why we had a hard time finding users of it.)

In providing an academic evaluation of these tools, it’s important for us to approach the field with a minimum of bias. I have a natural bias against tools that rely on security through obscurity, and it’s been hard for me to put this bias aside in evaluating circumvention tools. As it turns out, some of the closed-source tools developed by the GIFC group are some of the most impressive tools we’ve seen, a finding that’s reminded me to try to stick with an objective evaluation method and not with my preconceptions about the field. I don’t encourage the usage of tools I haven’t had the chance to evaluate, and as mentioned above, I’ve made clear to all interlocutors that I haven’t been able to evaluate Haystack. It is frustrating to me that I’m not able to act both as an advocate for or agains tools in this space and as a trusted evaluator – there’s a conflict between those roles that I’ve not been able to bridge and that limits me to speaking publicly about tools I have been able to evaluate.

I can understand Evgeny’s frustration with the popular press’s embrace of Haystack, and frankly I’ve shared that frustration. Had I praised Haystack, Evgeny would be well justified in calling me out. Evgeny feels that I’ve failed by not making public my concerns about Haystack before I evaluated it. Had I done so, Evgeny or any other commentator would have been justified in calling me irresponsible in rubbishing a tool without examining it, and critics of the testing methodology we’ve worked to develop at Berkman would have been right to ask questions about our objectivity as reviewers. Evaluating these tools forces my colleagues and me to be very careful about what we say regarding these tools, especially tools we have not been able to obtain information about.

I’m happy that the scrutiny of Haystack will lead to someone conducting a thorough evaluation of the code and protocol behind it. I hope that the justifiable anger over the press’s coverage of the tool will lead technology reporters to ask better questions before celebrating these tools unquestioningly, as my colleague Jillian York suggests.

19 thoughts on “Evaluating censorship circumvention tools and “irresponsibility””

  1. thanks for your response, Ethan – I’ll duly respond to it on my blog. One point I’d like to emphasize is that the bulk of my criticism related to the responsibility of INTELLECTUALS, NOT ACADEMICS, as the beginning of your post seems to imply. I’ll expand on that in my response.


  2. Indeed. The issue for me, at least, is that to the extent that I’m an academic focused on this topic, it can trump my ability to act as an intellectual in the space.

  3. Jake, I’m impressed by:
    – their wide usage
    – their performance (speed in loading pages)
    – their blocking resistance strategy, which has been quite successful in working against aggressive countermeasures.

    In our study, we gave Ultrareach high marks on performance and cautioned users about the fact that they have the capability of filtering or monitoring traffic – a problem common to most proxy-based circumvention solutions, and lower marks to Freegate, based on poorer performance and a data leak, which we informed the developers of. Hal and I remain in dialog with the developers and think they’ve done some excellent thinking about countermeasures to aggressive blocking – Roger and others at Tor have been involved with some of those discussions.

    Our study is at http://en.scientificcommons.org/51835899

  4. Pingback:   links for 2010-09-15 — contentious.com

  5. I haven’t seen any numbers of their users or for their performance – do you have a source for that?

    I’ve heard good stuff about their blocking resistance strategy – it sounds impressive in its brute force!

  6. Jake, we’ve used a combination of their numbers, descriptions of how many IPs they’re using and data from Google Ad Planner which helps us get a sense for how many people in China are searching for their software. None of this is perfectly verifiable, which we disclose in our report, but we’ve gotten to know the lead developer quite well and I think we have a sense for usage within an order of magnitude.

    Happy to brief you on their blocking resistance stuff. Email me, and we’ll talk. Indeed, brute force, but very elegantly deployed.

  7. Pingback: Haystack kaput/ Godard against intellectual property… A roundup… | Erkan's Field Diary

  8. Ethan, thanks for posting the link to that study (which led me to the very fine response by Tor). That’s exactly what I was looking for. Is there a set up for another test? I see a need for a constant independent analysis of these tools under different conditions (a sort of regularly updated consumer reports on circumvention tools, I guess). Perhaps that already exists, though.

  9. Drew, my colleagues at Berkman and I are planning a repeat of that study by the end of this year. I share your sense that we need an ongoing independent analysis of these tools – it does not exist, and I fear there are serious challenges to organizing and funding such a study.

  10. Pingback: WSJ: No Quick Fixes for Internet Freedom « Center for Innovation News Study

  11. Pingback: Should we oppose sit-ins just because crazy people can abuse them? : nice3.keyhome.info

  12. Pingback: apple1.iphonegreen.info » Should we oppose sit-ins just because crazy people can abuse them?

  13. Pingback: More on Internet intellectuals and the Haystack affair | nice3.keyhome.info

  14. Pingback: Should we oppose sit-ins just because crazy people can abuse them? | apple3.iphonegreen.info

  15. Pingback: Want To Work With Browne and Sharpe Tools For Your Projects? | Wind And Solar Power For Your Home

  16. Pingback: More on Internet intellectuals and the Haystack affair - baby1.ecobaby-home.com

  17. Pingback: More on Internet intellectuals and the Haystack affair

Comments are closed.