How a Pakistani ISP briefly shut down YouTube

One of the inspirations for Global Voices Advocacy, our online free speech project, was the good work being done by Don’t Block the Blog, a project led by two Pakistani bloggers to protest that nation’s block of Google’s Blogger platform. Don’t Block the Blog has been a real innovator in the anti-censorship space, and has done some excellent work reaching across borders, including the border with India, where they provided scripts to allow India’s bloggers – who were briefly blocked by their government – to reach their blogs. This sort of cross-border collaboration is what Global Voices is all about, and we hope to encourage more collaboration when some of these anti-censorship activists meet face to face in Budapest this June.

Today I’m leaning heavily on the Don’t Block the Blog guys to understand Pakistan’s attempts to block YouTube, and the widespread consequences of that decision. Sami ben Gharbia has a copy of the block order posted at GV Advocacy. It orders Pakistani ISPs to block a single URL – http://www.youtube.com/watch?v=o3s8jtvvg00 – as well as three IP addresses. The URL now leads to a YouTube page that declares “This video has been removed due to terms of use violation.” But many Pakistani users are discovering that YouTube, as a whole, is inaccessible as their ISPs are blocking access to the IP addresses associated with the site.

There are very few Pakistani ISPs sophisticated enough to implement the block of a specific URL – the Don’t Block the Blog guys note that ISPs using the TWA-1 cable rather than the FLAG cable to connect to the global internet can access all of YouTube, just not the specific URL ordered to be blocked. Most ISPs followed the block order much more closely, blocking the IP addresses and preventing Pakistani users from accessing any content on YouTube, not just the offending video. (There’s speculation that some ISPs chose to block widely because there are new videos on YouTube alleging fraud in recent parliamentary elections.)

While it’s always a drag when countries overblock, taking offline whole services instead of offending content, it’s become increasingly common. Sami’s Access Denied Map shows blocks – current or past – of YouTube in Turkey, China, UAE, Iran and other nations. What made the net sit up and take notice was that a Pakistani ISP managed to take YouTube off the net as a whole for a brief period of time.

Here’s how it happened, more or less:

There are at least three ways to block internet traffic to the customers of an ISP. The most exact way is to block specific URLs, like the URL specified by the Pakistan Telecommunications Authority. There’s two problems with this, from the perspective of the blocking ISP. It’s expensive – every request that goes over your network needs to be examined, not just at the header level, but within the packet to check for the offending URL. And it’s imperfect – tell the system to block “http://www.youtube.com/watch?v=o3s8jtvvg00” and it will fail to block requests for “http://youtube.com/?v=o3s8jtvvg00”, which will get you the same video.

A second way to block traffic is wicked cheap, but extremely ineffective – change the Domain Name System (DNS) records. People using that DNS suddenly discover that youtube.com no longer resolves to 208.65.153.238 but to an IP address controlled by the ISP – you can then put up a page saying, “YouTube is blocked by order of the Pakistan Telecom Authority.” The problem with this method is that it’s trivially easy to defeat – users can simply type in the correct IP address for the site, or switch to using an “unpolluted” DNS server like OpenDNS.

A third way involves blocking IP addresses. It’s far cheaper than blocking URLs, as the IP address is in the header of every packet, and all routers can be easily programmed to block IPs (it’s a routine way to fight denial of service attacks.) It’s effective – users need to use a proxy to get around it – though the major downside is that it’s overbroad. If you’re trying to block a single offensive page on Tripod, block the IP and you block all the millions of pages hosted there.

PieNet, a Pakistani ISP decided to block YouTube using the IP block method. (It’s unclear to me from these accounts if PieNet is part of Pakistan Telecom or not – I’d appreciate insights from my readers if you know.) They also wanted to implement a blockpage. Rather than tell their routers to block all traffic to 208.65.153.238 and redirect any requests to a blockpage, they decided to tell their users that they controlled all IP addresses that began 208.65.153.* – when a user requested YouTube, the DNS server would tell them that the site was at 208.65.153.238. Their browser would request that page, and they’d be redirected to a PieNet server, which would tell them they were blocked.

So far, so good. But PieNet’s rewiring of the net was propogated by their upstream internet service provider. PieNet accomplished its rewiring using Border Gateway Protocol, the internet’s core routing protocol. BGP is designed to allow large network operators (called autonomous systems, or AS’s) to announce paths to certain destinations on the internet. BGP is designed so that router prefer the most specific route possible. Google had announced a (correct) route that covered a fairly large block of IP addresses, including YouTube’s – PieNet announced a much more specific route, covering only 256 addresses. So routers that “trust” AS17557 – Pakistan Telecom – began telling their neighbors about a great new way to reach YouTube… which was now hosted by PieNet.

As a substantial portion of the world’s YouTube traffic attempted to make requests on PieNet’s servers, they went offline. In the meantime, Google released the correct routing information with more specific routing info, and most routers were looking at the correct address within two or three hours. To prevent Pakistan Telecom from propogating the bad address, its upstream provider, the Hong Kong based PCCW, removed Pakistan Telecom from the internet until they were able to fix its servers.

This sort of misrouting isn’t uncommon on the net – old net hands have been citing the AS7007 disaster, where a misconfigured router advertised that all Internet routes could be reached through it… and a good chunk of the Internet believed it and attempted to route itself through a single, overwhelmed router.

There’s at least two, and possibly three, things that went wrong in this case:

– Modifying routing records via BGP is probably a bad way to censor the net. (I’ll wait for confirmation from smarter people like Nart Villeneuve before removing that probably.) There are lots better ways to block IPs than to claim IPs that aren’t yours via BGP.

– Pakistan Telecom should not have advertised this route to other routers – they should have checked that PieNet was authorized to route this block of IP addresses before passing it on to the rest of the network. And PCCW should not have accepted Pakistan Telecom’s new route, as Pakistan Telecom didn’t have authority over these nunbers.

– Depending on who you ask, BGP is too trusting a network protocol. Back when the net was used mostly by academics, lots of systems simply trusted each other – you could send mail to another mailserver and ask it to deliver it for you. That doesn’t work so well in the age of commercial spam, and so network operators don’t do it anymore. The idea that one network can simply tell another that it’s got a route to certain IPs scares some operators, who see this as a system that would be very vulnerable to attack. There’s discussion on some blogs, mailing lists and message boards that this incident will get network operators to start using more secure forms of BGP that check carefully to make sure that another network is who it says it is, and that it has authority over the addresses it wants to route, before accepting its instructions.

Interesting to me has been the tone of discussions around the incident. Many network operators believed the incident was an accident “on the part of probably stressed NOC staff at 17557.” Others concluded that “it was almost certainly an intentional hijacking designed to make a political statement. A bogus BGP advertisement is a very loud and rude way to make such a statement.”

NANOG, the North American Network Operators Group, has a long and intriguing thread, which includes technical discussion on fixing the problem, speculation on how to prevent such accidents/attacks in the future… and political arguments about the Islamic world. At least one network operator threatened to block all routing changes from 17557 – Pakistan Telecom – while others pointed out that, even on “a sleepy Sunday”, it took less than two hours to fix this problem. In other words, even if a network operator wanted to “attack” the rest of the network by propogating bad IPs, they’d likely be shut down pretty quickly.

I’m always interested when realworld and network politics intersect. While issues like this take some effort to understand, the solutions offered and adopted on lists like NANOG can have profound implications for how the Internet works for everyone. It bothers me a great deal that there’s routine discussions on some forums on how to block entire nations from accessing a website. I’d be very disappointed to discover that some network operators began blocking routing information from Pakistan based on what, I suspect, was a mistake… but I wouldn’t be surprised.

At its heart, the Internet is based on trust. That trust has been badly abused over the past decade, and the net as we know it today is a patchwork of trust and verification. I don’t know who you are, but you’re able to read this blog entry from (pretty much) any IP address in the world. But when you go to comment, I try to verify that you’re a human being using a CAPTCHA… and Akismet tries to verify that you’re not a spammer by checking your comments against other comments identified as spam. This doesn’t always work – I end up blocking more comments than I mean to, but with over a million attempts to post spam on this blog in the last 18 months, running comments without verification would be disastrous.

What worries me is when we move from verification strategies to strategies that simply deny access based on preconceptions about national identity and character. Yes, it sucks that there’s a lot of spam and fraud originating from Nigeria. Yes, it’s a drag that Pakistan decided to censor YouTube. But it would be truly, truly stupid for network operators to stop accepting certain types of traffic from these countries due to laziness and prejudice.

Lots more information on the situation and the tech behind it at Data Center Knowledge and at Renesys. Apologies in advance for anything I’ve gotten wrong in trying to offer a readable gloss on what’s a very complex technical issue.