Mapping: Infrastructure and flow

I love airline route maps. I’ve fallen asleep staring at the tangle of possible journeys so often that I sometimes confuse the capilaries I see with my eyes closed with the red paths of Northwest flights hubbed out of Detroit and Minneapolis. I love the questions the maps raise: why is there a direct flight on Air Canada from Halifax to Fort McMurray in Northern Alberta? (Lots of Nova workers in the oil sands, I suspect, but I never would have asked the question without the map.) Why is Chengdu such an important Chinese air hub? Why does MIAT (Mongolia’s airline, affectionately known as “maybe I’ll arrive tomorrow” by regular customers) fly to Berlin, and no other western European cities? Does a direct Air Madagascar flight to Milan imply a strong Italian-Malagasy connection, or was Malpensa just one of the few airports where they could buy a landing slot?

These maps are deceptive in a way. They let you know what’s possible, but not what actually happens. The Northwest map will show you flights from Detroit to both Albany and Bozeman. While it’s good to know that it’s possible to get between those cities by flying Northwest, it doesn’t tell you how easy or difficult it might be to make that trip, how often those flights run, or how many people choose to make that trip. That’s okay – the job of maps is to tell a traveler where she can go, not where other travelers choose to go. But trying to extrapolate too much from a map of infrastructure may be a mistake – is the Ulaanbataar/Berlin link the sign of close governmental and trade ties between Mongolia and Berlin? Or an accident of history, airport capacity or other factors?

This lovely video gives a different picture from the route maps. It’s a simulation of global air traffic from the fine folks at the Zurich University of Applied Sciences. The map uses data from Flightstats.com, and overlays their position on a Miller cylindrical projection. Compared to some of the other flight data porn the folks at ZHAW have churned out – like their amazing Radar mashup of flights over Zurich, using live transponder data from aircraft – this was a pretty simple hack.

I’ve watched the video half a dozen times today, getting different insights each time. Popular routes become apparent – the arc of travel from the Northeastern US to London, Paris and Amsterdam runs west to east as night falls, and reverses as morning breaks. The popularity of that ocean crossing vastly outpaces traffic across the Pacific, connecting Tokyo, Manila and Beijing to Los Angeles, San Francisco and Seattle. There’s more traffic from Brazil to western Europe than I would have guessed, and virtually no traffic across the southern Atlantic or Pacific. Domestic traffic in the US, India and China, and intra-EU travel is vastly more common than trans-oceanic travel. As the US is covered with yellow dots representing airplanes, international travel looks like a rounding error in comparison to domestic flights.

It’s not a map you’d want to use in planning your vacation, perhaps, but it would be a useful one to turn to if you were tracking the spread of an epidemic, for instance. If you’re studying SARS, it’s useful to know that you can, theoretically, get from Guangdong to Johannesburg – it’s lots more useful to know that most of those travellers are heading to Hong Kong, Toronto and New York City.

It’s a map of flow, not of infrastructure. It reveals infrastructure – the location of airports, the preferred air routes followed – because they appear as bright spots, places where lots of flow originates. A map of infrastructure – a map of potentials – shows every airport as co-equal; a map of flow shows you which airports are heavily used, which are pivotal nodes in a network. If you’re an executive at a fast food company, an infrastructure map of highways is moderately helpful – it’s obviously wise to place your stores in places where drivers could theoretically reach them, rather than in the middle of a desert. (No one told Pacific Bell this, obviously, before they erected the legendary Mojave Phone Booth.) But a map of flow is what you really need, showing where drivers are likely to go, and where they’re likely to come purchase your grease-laden wares.

It’s hard to map flow. Infrastructure tends to stay put. But people, cars, and shipping containers move all the time. To build accurate maps, you can’t simply plot the location of an airport once – you’ve got to map each plane that flies during some period of time. Things that don’t stay put aren’t always happy about being mapped. In simplest terms, maps of flow are a form of surveillance. Mapping your personal “flow” – in the way that the BBC is tracking a shipping container around the world – would likely be a gross violation of your privacy, as it would probably reveal more about you than you’re strictly comfortable sharing.

My friends Sandy Pentland and Nathan Eagle have been experimenting with something Pentland is calling “reality mining“, using surveillance of individuals via their mobile phones to extrapolate information about social networks, individual health and events in the news. Eagle tells me that the system was so effective, it could determine which of the anonymous participants were dating, and was able to correlate behavior to events like the Red Sox World Series victory, during which cellphone users clustered in bars and crossed the river to celebrate near Fenway. Unsurprisingly, a lot of sponsors are interested in this research, including mobile phone companies and advertisers – it’s not unrealistic to believe that mobile phone companies might, at some point, offer you free basic phone service in exchange for your behavioral data (collected by tracking your phone) and the opportunity to target ads to you based on your location. (See Blyk, a free mobile phone service in the UK, targetted to young people and ad sponsored…)

The maps Pentland and others are making tend to make us the most nervous when we place ourselves in them as individuals. We wonder what a map of our actions will tell others. We’re generally more comfortable with them in aggregate. Leaving the Berkman Center, I look at Google Maps to see whether the traffic heading west on Route 2 or I-90 is lighter. This is a useful thing and I’m very glad that someone is monitoring road conditions and letting me make intelligent decisions about which way to drive. On some level, I realize that my beat-up black truck is part of the overall picture represented as a green, yellow or red line. But that map generally doesn’t make me uneasy in the way that a map that allowed you to click on it and see “1999 Toyota Tacoma, 27 mph, heading west on Massachusetts Ave, MA license plate 345 GDF”. The former reads to me as mapping of flow, the latter as surveillance, but it’s not entirely clear to me where the line should be drawn between the two ideas.

The map above is called “In Transit” and is part of the Cabspotting program run by the Exploratorium, using data from Yellow Cab and visualisations by the folks at Stamen Design. All yellow cabs in San Francisco are equipped with GPS and report their location to dispatchers, automatically, once a minute – they’re being surveilled so that dispatchers can respond to requests for cabs or deploy cabs to another part of town. In this visualization, those minute-by-minute accretion of data points are blurred into lines, showing the paths that cabs take. And these paths can reveal some interesting things about how people flow through the city of San Francisco.

Those who know San Francisco will immediately pick out the major highways – 101, 280 and 80 – and the paths across the Bay Bridge and the Golden Gate. It’s not hard to intuit where downtown is, to get a sense for the comparative popularity of various routes in and out of the city. The blank spots, on the other hand, are a little confusing. The area near #5 on the map is the Presidio, a former military base that’s now a park… which helps explain why there’s not much cab traffic through it. The areas just south of #4 and #7 aren’t parks – they’re Potrero Hill and Dogpatch, neighborhoods that are better known for industry and low-income housing than for tourist attractions or dot.com startups. To their southeast is a large blank patch on the map: Bayview and Hunter’s Point, a predominantly African-American neighborhood that surrounds a former naval shipyard. In other words, some areas are blank because there’s no good way to drive a taxi there. In other cases, they’re the neighborhoods where few people call for a taxi… or where the taxi drivers aren’t willing to go. The street map helps you figure out how to get from 3rd Street and Evans Avenue to Union Square, while the flow map makes it clear that you probably shouldn’t count on hailing a taxi to make the trip.

Maps of infrastructure visualize what it’s possible for people to do. Maps of flow show what they actually do. The two may diverge sharply.

A few years ago, if you wanted to send an email to a friend across the street in Accra, there’s a good chance the message would travel through the US or the UK on the way. Ghana had several competing internet service providers, and each provider bought internet connectivity from a different vendor. The vendors’ networks connected, just not in Ghana. So sending email across town meant sending a message on one ISP, to the US, transferring over to the other ISP, and back to Ghana, a journey that involved two satellite hops to cross the Atlantic. This is called “trombone routing”, and it’s generally something to be avoided.

If you mapped the network traffic of Ghanaian internet users – the flow – it sure looked like they were sending a lot of bits to and from the US. This might have been a result of trombone routing of emails between Ghanaians. Or it might have been because many websites are hosted in the US, and Ghanaian users wanted to read cnn.com, espn.com, etc. Knowing which it was mattered – if lots of traffic was local, it would make sense to construct an Internet Exchange Point (IXP), a crossing point for local ISPs to exchange traffic. If it was mostly requests to US webservers, the IXP wouldn’t save much money and probably shouldn’t be built. An infrastructure map would be no help – almost all traffic needed to go through the US, even if the intent was to communicate locally. To build a map of flow, Ghanaian ISPs would need to monitor their traffic, distinguish between domestic and foreign requests, share this information with fellow ISPs and make a decision regarding the utility of an IXP.

Ghanaian ISPs made the decision to build the Ghana Internet Exchange not based on understanding their own flow, but by looking at the behavior of other African exchange points. When ISPs in Johannesburg started exchanging traffic directly, they discovered that roughly 50% of their traffic was local to South Africa. The administrators who set up an exchange point in Nairobi saw roughly 25-30% local traffic. The disparity? There’s a lot more web servers hosted in South Africa than in Kenya, and hence more local traffic. To make the decision to build an IXP on a rational basis, you need to know not just the flow of internet traffic, but the flow the traffic would take if it were routed via an IXP. You need to know not just what users are doing, but what their intention is. This is a tough enough mapping challenge that you end up guessing, not analyzing.

The distinction between maps of infrastructure and maps of flow matters to me because I think it can help explain certain misconceptions and misunderstandings about our connected world. My contention – with very little to support it, frankly – is that we tend to assume more connections than actually exist. We see a map of infrastructure that shows it’s possible to fly from Antananarivo to Albania and assume, on an unconcious level, that the connection is routine, frequent, common. We look at maps of the internet – a near-worldwide tangle of undersea cables – and assume that data flows everywhere, connecting every one of us.

A map of flow would help us understand a more complicated reality. You can fly from Antananarivo to Albania, but you might be the only person this year to make the trip. Traffic flows between Ghana and the US via the Internet. We can see a cable – SAT-3 – that connects West Africa to the global internet through Europe and India. A map of flow could tell us whether that connection is symmetric, whether Americans are looking for information from Ghanaweb as often as Ghanaians are looking at ESPN or CNN. If we could see flow, we might detect the dark spots, the places reached by infrastructure but disconnected – through language, economics, or force of habit – from global flows.