The new Technorati: advertiser-friendly, foreigner-free?

A couple of years back, Technorati dropped out of my life. It was a sudden break, though I didn’t notice it at first. I blog using the WordPress platform, and WordPress relied on Technorati’s API to track mentions of my blog in other blogs, which I saw daily on my blog’s dashboard. And then WordPress began using Google Blog Search instead. I didn’t notice the difference for a while, and when I did, I didn’t really care. Google Blog Search, at that point, was pretty good, and it met my needs – it gave the the reassurance that people were reading and commenting on my words and that I wasn’t just wasting time talking to the ether.

And then Google Blog Search got less usable – first it got spammy, and then it got sparse. I turned back to Technorati and to Blogpulse and discovered that neither was especially satisfying. Talking with friends who blog, we agreed that it was strange and sad that there was no worthy blog search engine. In a meeting at Berkman yesterday, we were bemoaning the fact that Technorati had disabled their API, wondering whether this was a sign that the company was heading towards extinction.

I realized I hadn’t actually looked at the Technorati site for quite a while. I was surprised to discover that Technorati is back. It’s very different from what it was, and in some ways, much better. And, in one way that’s critical to me, it’s much, much worse. The site’s return raises some fascinating questions about the nature of the blogosphere, its influence and importance.

Self-obsessive that I am (a trait shared by the vast majority of bloggers), I checked Technorati to see how this blog ranked. When I checked Technorati regularly, this blog usually squeaked into the top 5000 blogs tracked by the site – in the past couple of years, I’ve slipped in influence (I’m sure you’ve noticed – thanks for not mentioning it) down to roughly 7,000. But I’ve now vaulted back to prominence with a ranking of #1116. Woo hoo! Except that I’m no longer 7,000 of 133 million – I’m 1,116 of 825,402.

Say what?

Technorati have historically been the cheerleaders of the blogosphere, pointing to an increase from four million blogs in 2004 to 70 million in 2007 and 133 million in 2008. Behind the scenes, people familiar with the challenges of indexing blogs knew that these numbers were suspect, in at least two directions. They were inflated, because the pingservers that aggregators like Technorati used to build their catalogs were riddled with spam. And they were undercounting the blogosphere, because many bloggers around the world – particularly those in China – use blogging platforms that don’t talk to pingservers, rendering those blogs invisible to ping-based catalogs. Dave Sifry would announce that there were 30 million blogs and proud internationalists like me would announce that the number was surely undercounting Chinese blogs, where the China Internet Network Information Center reported 47 million bloggers with 72 million blogs.

So what happened? Well, first, Technorati kicked out the non-English speakers. A quick tour through the top 100 sites indexed by Technorati reveals no non-English blogs. That top 100 list used to be quite diverse. I published a paper in Public Choice using data from Technorati in September 2005 that saw 15 of the top 100 weblog authors writing from outside the US, in Chinese, Italian, Portguese, Japanese and German. Some of those blogs have died, while others continue to be active and influential. Blog de Beppe Grillo is an incredibly important site to Italian political discussion – Alexa ranks it as 5,016 in the world in terms of traffic, 135 in Italy, and Google Ad Planner estimates 840,000 visitors a day, generating 11 million pageviews. That makes sense – Grillo is, in a very inexact analogy, Italy’s blogging Jon Stewart.

Technorati knows about Beppe Grillo. They just don’t think he’s very important. He gets a 1 in influence, the lowest rating on a scale from 1 to 1000. (I get a 611. Take that, you protest-leading, profanity-spewing, politically influential funnyman. That’ll teach you to actually reach an audience of millions!)

Other influential internationalists don’t make the index at all. My friend Harinjaka – one of the leaders of a blogging campaign in Madagascar, sufficiently influential to get invited to the TED conference – doesn’t appear at all. Others appear with a surprisingly low rank. Roland Soong’s indispensible EastSouthWestNorth – the most important blog for English-speakers trying to understand China – ranks 59,101, with an influence of 113. The “influence” score isn’t easy to understand anymore – it used to measure incoming links in the past six month. Now, “Authority is calculated based on a site’s linking behavior, categorization and other associated data over a short, finite period of time. A site’s authority may rapidly rise and fall depending on what the blogosphere is discussing at the moment, and how often a site produces content being referenced by other sites.”

In other words, links and some other stuff. Fair enough. But an algorithm that doesn’t see Beppe Grillo or Roland Soong’s influence has got something badly wrong with it. Or simply refuses to consider pages with substantial non-English content. (Soong’s blog is so important because it translates large volumes of text between English and Chinese, helping each group understand China-focused conversations happening in the other language.)

Technorati may have a very good reason for shrinking their catalog and kicking out the non-English speakers – it lets them build a carefully classified, hand-edited catalog. That blog directory is an extremely helpful resource, both to people who want to explore (English-language) blogs and to internet researchers. On the other hand, it’s hard to believe that the universe features only 88 basketball blogs, given that I can name 10 sumo blogs off the top of my head.

My guess is that Technorati’s good reason has to do with repositioning advertisers’ understanding of what blogs are and aren’t. In the early years of blogging, the goal was to convince tech pundits and financial markets that blogging was a real, important and growing phenomenon, so that investing in a blogging search engine sounded like a good idea. Now the goal is convincing advertisers that bloggers are creating highly-targeted, advertiser friendly content. That would explain why this year’s State of the Blogosphere doesn’t feature statistics about the growing population of bloggers (Now up to 830,000! Down from 133 million!), their geographic distribution (More bloggers in Japan than the US! And we don’t index them!) and focuses instead on a survey of 2,828 bloggers, asking them about their motivations for blogging.

Technorati classifies the new, advertiser-friendly 2009 blogosphere into four camps:
• Hobbyists (72%)
• Part-Timers (15%)
• Corporate (4%)
• Self Employeds (9%)

In other words, the criterion for classification is blogger’s financial motives. Part-timers blog to supplement their income, corporate bloggers sling bits for an employer, while self-employed’s labor in the bit mines on their own. Those of us, like me, who don’t actually make money from our blogs are “hobbyists”. We are evidently on the wane, due to “an increase of work and family commitments”, making those professional bloggers ever more important.

However loathsome you find this categorization, it helps explain where Technorati’s trying to go. Their business isn’t a comprehensive blog directory – it’s the hub of an advertising network that now ranks fifth in the universe of social media, managing ad inventory on 450 web properties. Persuading advertisers that bloggers are “are a highly educated and affluent group,” not to mention mannerly, neat and well scrubbed, recently helped the company raise another $2 million in venture capital funding.

And Technorati’s right – there has been a significant move towards professionalization in the blogosphere. Many of the top sites Technorati is tracking are highly professional, multiple-author ad supported newsrooms. Some of the bloggers who were primarily interested in sharing links or status updates have moved to Twitter or other tools better suited towards brief updates. Blogs have moved into longer-form essays and journalistic stories… though the hobbyists in the crowd, like me, might point out that they’ve also become platforms for academic publication and collaboration, for political organization in the US and elsewhere, for citizen media, whether or not those activities directly yield advertising dollars.

So here’s the new Technorati. It works better than it used to, and there are clean, well-lit pathways to almost a million blogs. What’s wrong with that?

Nothing, so long as people understand that Techorati doesn’t index the blogosphere. It indexes the blogs that it indexes, excluding those that don’t make sense in its new paradigm. And that’s got consequences. You may never have clicked over to Beppe Grillo when he ranked high in the top 100, assuming that since you didn’t read Italian or follow Italian politics, there was nothing for you on his site. In the process, you would have dicovered that virtually every post is translated into English, and that Italian political culture has a playful and performative quality that US politics would really benefit from understanding, if not embracing. At the very least, the presence of non-English blogs in that top 100 were a reminder that the Internet isn’t an English-only space, and that citizen media isn’t just a North American/Western European phenomenon. Technorati used to remind us that the Internet was a crowded, complicated, multiligual, multicultural place – now it tells us that the Internet speaks English and is safe for advertisers.

Perhaps the Technorati we’re seeing today is a preview of a larger transformation. Perhaps we’ll see language and culture-focused Technoratis, indexing Japanese, Chinese, Persian, Portuguese and Malagasy blogs. Maybe days four and five of this year’s state of the blogosphere will remind us of the global import of blogging, not just for advertising electronic gadgets, but for challenging coups and dictators.

And maybe not. We’ve got a little website called Global Voices that tries to make the rest of the blogosphere accessible to English readers… and, in turn, we now make that website accessible in dozens of other languages. We used to rely heavily on sites like Technorati to help us find bloggers in other parts of the world. Now we’ve got hundreds of dedicated new media people from Kyrgyzstan to Kiribati who’ll help you understand what people are talking about in those countries… whether or not Technorati chooses to include them.