Mapping the American Commute

Update, 20 September 2015: scroll to the bottom of the post if you want to download the data.

One of my summer projects this year has been attempting to map the American commute, following earlier work on a similar subject. Put simply, I've attempted to put together a map which shows commuting connections between locations in the contiguous United States, using the most fine-grained data I could find. Some of the results of this went into a recent piece in WIRED, and also CityMetric, and the larger piece of work it's based on is part of on-going research into the best ways of mapping commuting flows. The main images are below, followed by some more technical information. For now, all you need to know is that these images show commuting connections of 100 miles or less between Census tracts in the lower 48 states. You'll have to forgive me if your city isn't labelled! 

Higher resolution image available here

And now some zoomed in versions...

Zoom in of the west coast

Texas, and beyond!

Interesting patterns of connectivity in the Midwest

Look closely for some interesting inter-connections

The famous BosWash megalopolis

But this just shows where people live, doesn't it? Yes it does. But it also shows how the places where people live connect with other places from a functional economic point of view, at a fairly fine-grained level. It offers a slightly different view than just looking at the urban fabric alone which, I might add, is interesting in itself. Mapping flows like this is not exactly new, as this paper from Arthur Robinson (1955) on Henry Drury Harness (1837) demonstrates. Nonetheless, I haven't seen anyone map travel to work at this resolution for the United States, so I thought I'd have a go myself. 

If you spend some time looking at the big version of the map you can begin to see how places connect and where there are obvious disconnections, even between places that are not that far apart. One thing that you can pick up from the complete dataset (but not this batch of maps) is the growth of mega-commuting, as explained by Melanie Rapino and Alison Fields of the United States Census Bureau. 

Background information: the data I used is the most recent tract-to-tract journey to work dataset from the American Community Survey. This dataset covers journeys to work between the c74,000 census tracts in the United States and the complete dataset has around 4million interactions. I mapped this in QGIS, using methods I've described previously on this blog. The tricky bits were dealing with the messy FIPS codes, dealing with the size of the dataset, and trying to decide what to label. There is quite a bit of error in the dataset (as acknowledged by the ACS people) and each individual flow line has a margin of error value associated with it, from which I also calculated the coefficient of variation. This is explained in a more detailed working paper, which I expect to publish in the coming months.

Update, 20 September 2015: there has been quite a bit of interest in the underlying dataset I put together to create the maps, so I have decided to make the whole shapefile available here in the hope that others will find it useful and be able to produce some interesting analysis or visuals from it. I'm hoping someone will do a cool interactive web map of it, but it might be quite technically challenging. If you do use it, make sure you read the associated working paper, which explains the process and the underlying data. One word of warning: the uncompressed file is pretty big so you'll need a good computer.

Mapping the American Commute: download the data (213MB, zipped shapefile)

Mapping flows in ArcGIS

This short post is about the process of flow mapping in ArcGIS and not really about the end results - though the maps are quite interesting. I've done quite a bit of flow mapping in the past and am now getting ready to work on the next set of Census flow data in the UK (which should be out in November) so I've been experimenting with some tools. I've written about this in the past in papers in Computers, Environment and Urban Systems and also Environment and Planning B but those papers are a bit long-winded! Other people have produced beautiful flight path maps so I thought I'd experiment with the same data using the relatively new ArcGIS XY to Line tool in version 10.0 (it can be found in ArcToolbox - Data Management Tools - Features - XY to Line at the bottom of the list of tools). For more on other methods and previous iterations of this kind of thing take a look at the work of Nathan Yau, Michael Markieta or James Cheshire.

For anyone wanting to map flows in ArcGIS, Michael Markieta's tutorial is probably a good place to start but be prepared for things to go awry in ArcGIS... When I mapped the 59,000 or so flight paths in the map above using XY to Line and one single dbf file (or csv, etc. - it makes no difference) the resulting shapefile only contained 16,066 rows. This happened every time I tried it and a couple of times my shapefile had only 73 rows. Another time it had ~14,000. That's why Markieta recommends splitting the file up - although I just cut it up into chunks of 16,000. Interestingly, I ran into exactly the same problem with my CEUS paper a few years back using Glennon's flow data model tool - though the limit was about 32,000 before it cut off. 

Another very annoying feature of XY to Line (for me at least) is that when you choose the Great Circle option under 'Line Type' it takes much longer to compute and the resultant shapefile is enormous. The shapefile for flight paths in the above map is about 10MB whereas the great circle version was over 450MB for one 16,000 chunk alone. Not sure if anyone else has run into this but it doesn't seem like a very efficient way of doing things! [Edit - as @baeing has reminded me, it's because shapefiles don't support curves - though geodatabases do.]

Once I had my complete shapefile I moved to QGIS, added in a world layer from Natural Earth and then experimented a little with symbology. I also experimented with different styles and projections to produce some of the maps below. That's all for now - I just hope ESRI are able to improve upon the current version of XY to Line because when it does work it is really fast (on my machine at least) and straightforward.

Very similar to above, minus text

Short haul, different projection

Short haul, different projection, borders

Slightly different symbology

And, yes, I know that flights from Australia or New Zealand typically go over the Pacific rather than the long way round! I'm just showing the XY to Line outputs as they are in this post.

Everything is Connected

One of the things I said I'd be blogging about is urban issues and things to do with urban life, specifically as they relate to planning. However, I thought it would be good to do a quick post on the topic of connections and how it's really impossible to fully separate the urban from the non-urban or rural. This is a topic which a colleague of mine, Andreas Schulze Bäing explored in his PhD thesis and one which I do research on, but without so much focus on the rural dimensions. I'm also aware that I come from an area that is not exactly an exemplar urban area (the Highlands of Scotland), even though I've lived all my adult life in cities (Glasgow, Columbus (Ohio), Liverpool). So, if you're from a rural area like this...

... it doesn't mean that cities are irrelevant. In fact, given that they are now seen as the drivers of economic growth in many advanced industrial economies, we really should be aware of how things are connected. When you look at the major migration patterns in the UK, these kinds of connections are more obvious.


When you look at the patterns in even more detail, in this 3D image I created for migration patterns in the North West of England (high in-migration areas in red, out-migration areas in blue), we start to get a better understanding of the ways in which cities import people (mainly students and young people). However, they do export families who generally tend to rely on urban areas for a living but don't want to live there... But that's another matter.



So, everything is connected to everything else. This is exactly what Waldo Tobler's First Law of Geography states and I am a firm believer in it. Nothing should be studied in isolation (especially in planning) and that's why much of my work includes a spatially dynamic element. The relative importance of this connectivity may vary, but it is invariably important. We need to see the big picture if we are to understand what's going on in our own back yard, so to speak.