Mapping flows in ArcGIS

This short post is about the process of flow mapping in ArcGIS and not really about the end results - though the maps are quite interesting. I've done quite a bit of flow mapping in the past and am now getting ready to work on the next set of Census flow data in the UK (which should be out in November) so I've been experimenting with some tools. I've written about this in the past in papers in Computers, Environment and Urban Systems and also Environment and Planning B but those papers are a bit long-winded! Other people have produced beautiful flight path maps so I thought I'd experiment with the same data using the relatively new ArcGIS XY to Line tool in version 10.0 (it can be found in ArcToolbox - Data Management Tools - Features - XY to Line at the bottom of the list of tools). For more on other methods and previous iterations of this kind of thing take a look at the work of Nathan Yau, Michael Markieta or James Cheshire.

For anyone wanting to map flows in ArcGIS, Michael Markieta's tutorial is probably a good place to start but be prepared for things to go awry in ArcGIS... When I mapped the 59,000 or so flight paths in the map above using XY to Line and one single dbf file (or csv, etc. - it makes no difference) the resulting shapefile only contained 16,066 rows. This happened every time I tried it and a couple of times my shapefile had only 73 rows. Another time it had ~14,000. That's why Markieta recommends splitting the file up - although I just cut it up into chunks of 16,000. Interestingly, I ran into exactly the same problem with my CEUS paper a few years back using Glennon's flow data model tool - though the limit was about 32,000 before it cut off. 

Another very annoying feature of XY to Line (for me at least) is that when you choose the Great Circle option under 'Line Type' it takes much longer to compute and the resultant shapefile is enormous. The shapefile for flight paths in the above map is about 10MB whereas the great circle version was over 450MB for one 16,000 chunk alone. Not sure if anyone else has run into this but it doesn't seem like a very efficient way of doing things! [Edit - as @baeing has reminded me, it's because shapefiles don't support curves - though geodatabases do.]

Once I had my complete shapefile I moved to QGIS, added in a world layer from Natural Earth and then experimented a little with symbology. I also experimented with different styles and projections to produce some of the maps below. That's all for now - I just hope ESRI are able to improve upon the current version of XY to Line because when it does work it is really fast (on my machine at least) and straightforward.

Very similar to above, minus text

Short haul, different projection

Short haul, different projection, borders

Slightly different symbology

And, yes, I know that flights from Australia or New Zealand typically go over the Pacific rather than the long way round! I'm just showing the XY to Line outputs as they are in this post.

Why I Like 3D Maps

I do quite a bit of spatial analysis and mapping in my academic research, and some of it ends up on this blog. Over the past few years I've done quite a few 3D* maps - most recently one of population density in China. A comment by map-guru James Cheshire made me think about the 3D issue, hence this post, which attempts to say a little more about why I like using the third dimension, as it were. Also, there's not much about this stuff online at present. The two images below show population density in Europe at NUTS3 level, with a colour scale running from red (high density) to blue (low density). Click the images to enlarge.

*N.B. Data for some parts of Italy, Germany, and the UK are missing, 

but that doesn't matter for now - this is just an example.

The reason I like the addition of the third dimension with this kind of dataset is that you can tell more about the differences between areas within the same statistical category. Essentially, it adds an additional dimension of information that you can not observe from the conventional 2D map above. This is particularly true of the most high density areas in the first map. There is of course an issue here about the relative size of areas and how this might change the population density of different places but that is a different matter since I'm not in control of NUTS3 definitions! For more on this kind of thing I'd recommend looking at Stan Openshaw's work and for more on the utility (or futility?) of choropleths generally Tobler (1973) is an excellent starting point. Gale and Halperin (1982) is also worth a look.

If we assume that the main purpose of a choropleth map is to present and discover spatial patterns then there is sometimes a strong case for using the vertical dimension and extruding polygons using a z-variable (I do this in ArcScene, in case anybody is interested). However, there are some complications and I don't think it is always appropriate to go 3D. For example, depending upon the spatial structure of your data extruded polygons in one area can obscure those in another. There is also the issue of the different size of areas and the way these might have an impact upon the level of extrusion - i.e. if we used 1km cells for the European population density map it would look rather different in 3D - though this is possibly another artifact of the modifiable areal unit problem, as described by Openshaw. I've patched together three different examples from my blog in the image below, just for comparison. Another option would be to follow the example of Ben Hennig and produce population-weighted cartograms.

When I produce these 3D maps (or visualisations) I'm not trying to create a geographically precise rendering of space but rather I'm attempting to draw attention to variations in a dataset in a way which 2D maps can only do to a limited degree. They are abstractions and simplifications but in terms of understanding the world I find it can be an improvement. There is a little bit about it in this Environment and Planning B paper I wrote but I plan to write more about this in the near future (the'near future' in geological terms of course). 

*Also known as 2.5D in the GIS world, but I'll put that to one side for now...

Google Fusion Tables

A short post today about the wonderful Google Fusion Tables. It may sound baffling, but Fusion Tables are basically a service (provided by Google, obviously) that allows you to manage and map large databases 'in the cloud', as they say. In plain speak, you can upload large tables of data and display them and map them and share them. For someone like me who is into mapping data, this is great because all you need to do is get your data into kml format. This can be done with normal GIS software like ArcGIS but the best way I have found is to use shpescape.



The example above is just something I created a couple of years ago in ArcGIS and then recently uploaded to Fusion Tables. It's a 'tile finder' for Ordnance Survey data so you can figure out which 20km tile you need to get based on which area you are interested in. This is not the point here though - I'm just showing how you can get data into google map format very easily.

If you work with lots of data on a regular basis - and particularly any mappable data - you should take a look at Fusion Tables...

Repeating Shapes in ArcGIS

It's been a while since I wrote about a GIS tool. Sometimes when I'm working with geographic data the need arises to create a set of repeating shapes and I look for tools to automate it. In ArcGIS, the best I've found is Repeating Shapes for ArcGIS by Jeff Jenness.

Very easy to download and install and then you can get started. Some example images below based on hexagons and the documentation pdf.

How to Shade Your Maps - Colorbrewer

A short post today about 'Colorbrewer'. What is it? It's a website by Cynthia Brewer, Mark Harrower and Penn State University (home of many GIS/visualisation boffins) which you can use to help you decide what colours you maps should be. This is not a trivial matter and it is often difficult to develop an appropriate colour scheme that looks good, is colour blind friendly, print friendly and photocopy friendly. Hence the Colorbrewer website.


You can choose from a variety of colour schemes, you can choose the number of colours in you colour scheme, you can export the information in various ways and you can even download a 'colortool' extension for ArcGIS.

There is so much advice on the website - it really is excellent but I just never got round to blogging about it until now. Ultimately you still have to make choices yourself but with Colorbrewer these choices can be much more appropriate. Take a look...

Save as DBF in Excel 2007

One common problem for ArcGIS users over the past few years has been the decision of Microsoft not to allow users the option of saving files in dbf format from Excel 2007. It's not a problem if you are still using Excel 2003 but if you only have Excel 2007 it makes what was previously quite simple a lot more difficult.


One tool I've recently used is an Excel Add-In produced by Gyula Gulyas from theXLwiz blog. There are some useful dbf videos on his youtube channel which is, possibly, the only one in the world to combine videos about dbase file formats and videos of cheerleading!

Anyway, I have the tool. It cost me £17.34 ($25 USD) but it has been worth every penny. I know there are other options and I've tried some of them, but the Gulyas tool is so simple and effective and results in very compact file sizes. I highly recommend it!

Video Tutorials for GIS

I've not been saying much recently about screencasting, or video tutorials, but I have been working in this area quite a lot over the past year - mainly in conjunction with SalfordGIS. I've just come back from the Planning Research UK Conference in Chelmsford where I was speaking about my work in this area. So, I thought it was about time for another GIS screencast post.

The video below is one of a series I've produced in collaboration with SalfordGIS. This one is a basic ArcMap demo but we also have lots of MapInfo videos too. For both GIS applications skills from basic to advanced are covered. The videos are all around 3 to 5 minutes long each, with one task demonstrated per video. The idea is not to replace traditional GIS training but to provide quick refreshers of key skills.

Hawths Analysis Tools

Short post on a very useful GIS extension today, for ArcGIS 9.x. It's called Hawths Analysis Tools for ArcGIS and it is great. For example, if you have a layer with a number of different points, you can quickly create a n x n distance matrix in csv format. There are lots of other very useful tools. Here's an example screenshot:


















It really is great, and it's a FREE download.

Still working on things

January has been pretty quiet on the blogging front, but that's because I've been working on papers. One of these is in revision and the other is a new piece of work based on my PhD writings (but it's taken on new life too).

Anyway, more on the former. I'm attempting to revise a paper on flow mapping and to make things more interactive and more effective so I decided that I would build a small site in google sites and then put links to content there. So, if you're at all interested in flow mapping, kml, google sites, GIS, spatial interaction or anything closely related to any of these it may be of interest. If not, it does look quite nice.

There's an introductory page, some information about the data, a few colour maps, direct links to kml files overlaid onto google maps and some links to the KML files on my web space. There's also a few words about me and about the project that all this came from. As you can see, I've also added a slightly psychedelic banner. Link to site here.


In future, I plan to blog a lot more on non-flow-mapping-related issues. It's just that for the past year or so this has been a big part of my work. That's all for now...

GIS Screencasts

A quick post about my ongoing work with screencasting and geographical information system (GIS) software. Screencasts are video demonstrations of computer activity. Since teaching GIS is a very hands-on thing, screencasting is an excellent way to reinforce and enhance learning. I don't just use screencasting for GIS though; I also use it to demonstrate any kind of software or even for showing how to download material from the internet for different modules I teach at the University of Manchester. Here's a small animated gif showing the kind of thing I'm talking about:

Software like ArcGIS can be quite intimidating to new students with no experience of GIS so I find that the screencast approach is particularly effective. I'd like to think it will be much more widely used in higher education and in the teaching of GIS in particular within the next few years. The software is not difficult to learn (I use Camtasia Studio 5 but there are lots of other good tools out there - see this). That's all for now!

Getting Nerdy?

I'm always thinking of new ways to do new things and now I've got to the stage where I think learning some new tricks would be useful for my GIS development. I've started off trying to learn Visual Basic for Applications with ArcGIS so that I can develop solutions to problems I have or might come across in the course of my work. I've used the work of other developers in the past and I think it's now time for me to try and develop some tools of my own. I don't imagine I will get very good very quickly since I don't have much free time but I am keen to get going. We'll see how I get on in due course...

I've put a widget for this blog on my personal pages at the University of Manchester, which Alex Hardman pointed out to me. He's already nerdy but then it's all relative I suppose. Other than the VBA thing, I'm working on my screencasting project which I hope will end up being really helpful to students who need to use software in their courses but get stuck at some stage. This is a work in progress but things are going well so far.