The beating heart of the City of London

I've had a rush of blood to the head so here I am with a second blog in two days. I'm getting some slides ready for tomorrow's Modelling World 2015 talk in London, which is all about visualising mobility (see below) so I wanted to add in a couple of new visuals on commuting in and out of London. Visualisation can often be just a lot of fancy graphics. This can be useful in itself for a number of reasons (e.g. capturing attention on an important issue, drawing attention to unusual patterns in a dataset) but since I've been working with commuting data in England and Wales I wanted to focus on flows into and out of the City of London. 

This interests me for a number of reasons, including i) commuting can play a significant role in wealth creation and it also needs to be understood in relation to how we measure GVA; ii) commuting is often very stressful and damaging to the individual - particularly long commutes - so I'm interested in the kinds of distances involved and this can be seen easily on a map; iii) commuting can often be environmentally damaging - though this isn't what I'm mapping here; iv) commuting in and around London is often about green belt hopping so I was curious to see how much commuting comes from beyond the metropolitan green belt; and v) commuting is a two-way process and affects places at both ends and in between due to travel. 

So, here's what I did. I took the MSOA-level commuting data for England and Wales (table WUEW01 here), used a bit of QGIS, extracted frames from QGIS using the MMQGIS plugin, then patched it all together in GIMP to create an animated gif. One for inflows, one for outflows and one for in and outflows (thanks to Ebru Sener for the idea). It might run a little slowly in the blog post in a browser but see below for the images. Just to clarify, I've only shown flows of 25 or more into the City of London. Those not familiar with the data should be aware the the 'City of London' refers to the small area in the centre of London and not the entirity of Greater London! An obvious point but one worth repeating in case anyone is confused. A Greater London map would have many more data points, covering most of England.

Commuting flows (>=25) into the City of London

Same as above, but going back the way

The 'pulse' of the City of London

You should be able to get a better view of the images by clicking on them individually and if you want them to work more quickly try saving them to your own machine.

Can Google search data predict an election victory?

Today seems like a good day to write a little blog post on what search data can and cannot tell us. Why? Because of the story below, which has been on the front page of the MailOnline for much of the day. This is just the way news works, but I thought it would be useful to give a bit more information here. The story behind the story goes something like this...

Simon Rogers, datajournalist and Data Editor at Google in San Francisco, got in touch some time in April to ask if I could help him map party leader search patterns by constituency. I'd been doing a lot of work with search data for housing markets anyway so this seemed like an interesting idea. We took search data for all points that were geocoded (there were about 5,000 total across the UK) and then produced a constituency version for all 650 seats. The final constituency results matched very closely the proportions for the individual places. The data are for the previous 12 months. 

MailOnline front page 07/05/15

The big question is what this all means. Do I think, as the MailOnline suggest, that 'Google Search tips Cameron to win election'? No. Do I think it disproves it? No. Do I think the large volume of search for Nigel Farage indicates his level of popularity across the country? Again, no. However, it could indicate that people are more likely to show interest in UKIP in an environment when nobody else is watching or listening. But we don't know. Does this prove that Miliband will come third? Definitely not. The map merely indicates who was the most searched for party leader in each constituency. The intent and sentiments of individual users are not known. In my own research in housing market analysis I've tackled this by doing interviews with website users but since this data is from Google they could of course add in other terms which people might combine with party leader names (some more favourable than others!). 

Kate Newton from Bing also got in touch to say they had worked on something similar (though much more sophisticated) in relation to the Scottish Independence referendum last year. More widely, there is a body of emerging research (including my own) which looks at search patterns and subsequent activity - mostly in the field of economics. The results suggest that search can be analysed meaningfully to predict future activity. But that's not what the party leader piece was about - not from my perspective. Thankfully, other media outlets were more measured in their analysis - such as BuzzFeed UK, The Scotsman,  and The Telegraph. 

MailOnline story 07/05/15

What's most interesting to me? Well, I'm most interested to see how the search patterns relate to outcomes in key marginals. I suspect there will not be much of a pattern but if there is it will be interesting to attempt to take this little piece of work further - perhaps for the US 2016 election. Other than that, this is an interesting stocking filler on a day when the papers and TV crews are forbidden from reporting anything really substantial until the polls close at 10am. In the meantime, my favourite snippets from the search map...

David Cameron is the most searched for leader in his own constituency, but he's surrounded by a sea of purple, plus a blob of red and orange.

Witney - David Cameron's constituency

Perhaps a little predictably, there is a lot of search for UKIP in Kent but - strangely - it appears that in the constituency where Nigel Farage is standing (Thanet South) the party leader most searched for is Ed Miliband.

UKIP - lots of search in Kent, but not to much in Thanet South

The search results produce some interesting results. The image below is a good example. Natalie Bennett (Green Party) is the most searched for leader in Durham North West and next door in Durham North the leader (by search over the past 12 months) is Leanne Wood (Plaid Cymru). I suspect this was down to a localised spike in interest after the leaders' debates.

Durham - Green and Welsh Nationalist stronghold?

Other interesting nuggets to emerge were the way in which geographical patterns sometimes reflected the opposite of what you'd expect. The most obvious example was where Nicola Sturgeon (SNP and not standing in this election) was the most searched for party leader in several English constituencies, such as Chesterfield (below). Her excellent performance in the leaders' debates probably led to a spike in interest. Perhaps the SNP ought to consider putting up candidates in England too.

The SNP take Chesterfield? Not so fast.

A kind of similar situation to the SNP/Chesterfield example can be seen in the final image below, where Nigel Farage is the most searched for leader in Aberdeen North. This Scottish constituency has no UKIP candidate and, even if it did, they would be a long way away from the top party.

What's my prediction for the outcome of the election? The only prediction I'll make is that the results will look nothing like this map!

Flow mapping with QGIS

[Now updated with sample data file - see Step 1.]
I've written quite a bit about flow mapping with GIS in the past, including on this blog, and in a couple of academic papers. Previously, I'd used ArcView 3.2, ArcGIS 9 or 10 and MapInfo. MapInfo in particular has been my 'go to' GIS for mapping large flow matrices, thanks to a very short line of MapBasic code explained to me by Ed Ferrari. Others, such as James Cheshire, have used R to great effect, but this post is instead about flow mapping with QGIS, which I am extremely impressed with for its flow map capabilities. I've posted many of my QGIS flow maps on my twitter but in this post I want to explain a little bit about the method so others can experiment with their own data. Here's an example of a flow map created in QGIS - though in this case it's not a very satisfying result because of population distribution, county shape and so on*.

US county to county commuting

So, to the method. If you want to create these kinds of maps in QGIS, it's mostly about data preparation. I should also add that I currently use version QGIS 2.4 but I believe the method is the same in any version. Here's the ingredients you need.

1. A file with some kind of flow data, such as commuting, migration, flight paths, trade flows or similar. There should be columns with an origin x coordinate, origin y coordinate, destination x coordinate, destination y coordinate, some other number (such as total commuters) and any other attributes your dataset has (such as area codes and names). Here's an example csv file of global airline flows, if you want to experiment - it's the one from the screenshots below. I put it together using data from OpenFlights - by combining the airports.dat and routes.dat files. 

2. Once you have a file with the above ingredients, you then need to create a new column which has the word 'LINESTRING' in it, followed by a space, an open bracket, then the origin coordinates separated by a space, followed by a comma and a space, then the destination coordinates separated by a space and then a close bracket - as you can see below. You don't actually need to call the column 'Geom' as I have below, but when you import the file into QGIS it will ask you which column is the 'geom' one. You can create the new column in Excel by using the 'concatenate' function. If you're not familiar with it, there are loads of explainers online.

This bit probably takes the most time

3. Once you have your data in this format, you need to save it as a CSV so it's ready to import into QGIS. From within QGIS, you simply click on the 'Add Delimited Text Layer' button (the one that looks like a comma) and then make sure your settings look like the example below.

Make sure you click the right import button

Import CSV dialogue in QGIS - should be on WKT

4. Once you've done this, you simply click OK and wait a few seconds for QGIS to ask which CRS (coordinate reference system) you want to use. Select your preferred option here and then wait a few more seconds and QGIS will display the results of the import. You can then right click on the new layer and Save it as a shapefile, or your other preferred format. In the screenshot example above, the file with c60,000 airline flows took only about 10 seconds to appear on my fairly average PC running 64 bit Windows 7. I also tried it with 2.4 million lines and it only took about a minute. If you try this in ArcGIS - in my experience - it normally doesn't work with that many flows but MapInfo will handle it okay, but take longer. However, QGIS will render it more nicely as it handles transparency in a more sophisticated way and with hundreds of thousands of flows you usually have to set the layer transparency to 90% or higher.

The results, once you've done a bit of symbolisation and layer ordering, will look like some of the examples below.

Rail flows

All commuter flows

Bus flows - no labels, obviously

* I'm still trying to make sense of the US county to county flow map. The spatial structure of the counties and the distribution of the population make it more difficult to filter, so the above example is just a very rough (and not very satisfying) example.

Addendum: since a few people have asked, I've done a new post on how to make the lines appear to glow

A national map of cycling to work

I've recently being doing some visualisation work with the newly released Census commuting data from 2011. I've produced maps of all travel to work, and travel by car, train and bus. I've now done a map of cycling to work (below). This map is particularly interesting in relation to the patterns it reveals but also in relation to the strange long-distance flows we can see. I'm certainly not saying that anyone actually commutes by bike between Manchester and Bristol, as the map may suggest. Click on the big version and have a look around to see if you can spot anything interesting or particularly unexpected. A version with some place name labels can be found here.

Go to the full size version

This data comes from Question 41 of the 2011 Census form, which asked people to say how they 'usually' travelled to work in relation to the mode of transport which accounted for the largest part, by distance, of their journey. The results can look quite beautiful on a map, but they can also be confusing. Look closely at the map above and you'll ask yourself why there are so many long distance cyclists in England and Wales. More seriously, you might begin to question the validity of the data, the honesty of respondents or some other aspect of the results. 

The ability to interrogate datasets in this way is one of the strengths of visualising large datasets in that we can often immediately identify anomalous patterns or results that confound expectations or are just plain wrong. I'm not entirely sure what's going on with the long-distance flows. Perhaps some people take their bike on a train so ticked the 'bike' option, despite the train journey being longer. Perhaps some people live in one part of the country during the week and cycle to work there but then live at their usual address during the weekend and this is registered as their residence on the Census forms. I'm only speculating but this could be one possible explanation. 

In the image below, I've filtered the data so that only flows of 2 or more are shown. This significantly reduces the visual clutter, but also draws out stronger long distance connections between places such as Bristol and Manchester, and indeed Manchester and lots of other places. Take a closer look by clicking the link below this map. I've added some place names to this map to help with orientation.

Go to the full size version

I'd be keen to hear different interpretations on the data. You get similar results when you map the 'walk to work' data so there's definitely something interesting going on with how people have answered the Census question and the data we have to work with. I'm certainly not saying it's 'wrong', more that we need to understand what exactly it tells us. For now, I'll leave it at that.

N.B. Why didn't I include Scotland and Northern Ireland? The data are not out yet. It's not some ploy to exclude anyone and I know the blog title says 'national' so forgive me if that threw you. I intend to expand the analysis in due course.

HS2 geodata - for download

Yesterday I wrote a short post on the Guardian's Datablog about my difficulties getting hold of the route data for the proposed routes for the new high speed rail lines in England. Coincidentally (or maybe not) HS2 responded to my request at almost exactly the same time as the piece appeared online. Anyway, sometimes it does take time for public bodies to respond to requests so my real question was why the shapefiles were not available for download, given that they are available under the Open Government Licence. I have a few ideas about why this must be but it would be good to have some information from this on HS2, though maybe they're too busy with other things! Clarity on this issue might, however, reduce the likelihood of data conspiracy theories and enhance transparency.

Anyway, enough about that. So that other people don't have the same wait as me to get hold of the GIS data I've made them available here via the link below. A few important points to bear in mind...

1. The Phase 1 (London to West Midlands) route is the 'post-consultation' route from January 2012.

2. The Phase 2 (Leeds and Manchester) routes are the 'initial preferred routes' from January 2013.

3. There is an interactive map of the Phase 1 route on the HS2 web pages, which is quite useful.

4. Users of the data need to remember to acknowledge the source.

5. It's not my data - I'm just making it available.

6. You can also get these files from Barry Cornelius, but not - as yet - from data.gov.uk

7. The route data available for download here doesn't necessarily reflect the precise location of where the train lines will be built - particularly for Phase 2. 

HS2 shapefiles, as of 2 May 2013

Unemployment in Europe (via Google)

Despite recent headlines about data capture, Google remains an excellent source of (or gateway to) information on socio-demographic data. For example, if you type in 'population' followed by a country name, such as 'mexico' then this is what you'll get...

If you do this with any country you'll get the latest results plus a little graph which you can then click on and explore further. Similarly, if you type in 'eu unemployment' you will see a little chart showing EU unemployment - currently 10.2% for March 2012 - and how it has changed over time. If you click on the small chart you'll then see data for Europe and be able to add in data for other EU nations by clicking the boxes to the left. You can even embed this in a web page, as you can see below...

Apart from being convenient and accurate, this is also a very useful analytical tool when you need quick comparisons, like in the example below where I've compared Spain, the EU, Germany and Austria. As you can see the time-series data does not always extend as far back as we'd like but it is a great way to get your head round what is happening in different places without much effort at all. You'll notice in the embedded graphs that if you hover over a line it should tell you the data value for that point.

I've now changed the criteria in the chart so that it only includes unemployment for those aged less than 25 - and I've added in the UK too. This makes pretty grim reading for the EU, and Spain in particular...

This method also works for lots of other kinds of data. For example, if you type in 'us gdp' you'll see the data for the US but also have the option to add in lots of other comparators. One of the most interesting comparisons is looking at GDP over time, as you can see below.

I'm going to a conference in China at the end of June, hence my interest in national comparisons. This kind of thing has of course been covered extensively by Hans Rosling, but not many people know that it is fully integrated into Google's basic functionality.

Mapping Flows - An Update

In many previous blog posts I talked about my experiments in flow mapping (exhibit 1, exhibit 2 ...). Usually this was about migration or commuting data and in the course of writing an academic paper on the subject I also put together a small website about flow mapping, with some examples. Now I've done a follow-up paper to this which is just out in Environment and Planning B (a couple of extracts shown below).

The point of this work is not simply to make pretty pictures. That might be an interesting by-product but it is more about the process of taking data and giving it some kind of meaning by mapping it. This is by no means trivial when you're looking at migration or commuting patterns which link hundreds or even thousands of places.

With this much data, you often have millions of individual cells of data, so making sense of it can be impossible without some kind of visual approach - and that's really what the paper is about. It's not really very complicated - at least, not conceptually - but the power of this type of approach is in its ability to generate knowledge from raw data. 

In short, then, this kind of work is in many ways guided by Kenneth Boulding's maxim that "knowledge is always gained by the orderly loss of information".

The New ONS Website

Not all that long ago, the Office for National Statistics unveiled their new website which, in my view, is significantly better than the old one (screenshot of the new site below). There have been some teething problems but I just wanted to highlight some of the interactive data features.

There's a box to the right of the home page which says 'Interactive Content' and if you click on it you will be taken to a page full of more interesting visualisations of ONS data. There's a map and data visualisation using unemployment benefit claimants plus lots more.

On the population page, you can also see up to date snapshots for the UK, including the up to date total population (currently estimated to be 62.3 million). The site is not yet perfect but is definitely worth a closer look.

A Vision of Britain Through Time

I've been doing some work recently which involves the analysis of population change in England from 1801 to 2009. The data for this has come from the Census (which goes back to 1801) and mid-year population estimates (for 2009). The data itself has been sourced from a wonderful website called A vision of Britain through time, based at the University of Portsmouth. It's an innovative and pretty comprehensive exercise in historical stats and historical GIS.


As you can see from the screenshot above, you can enter a place name to find a place, you can look at various landcover maps, you can look at historical maps, Census reports and even travel writing by such distinguished contributors as Samuel Johnson, Daniel Defoe, Celia Fiennes and the wonderfully named Gerald of Wales.

Samuel Johnson's writing is from his 'A Journey to the Western Isles of Scotland' and is both shocking and funny seen from today's perspective. He very much disapproves of the 'Highland manners' and he calls Loch Ness 'a very remarkable diffusion of water without islands'.


The maps (as above) are really interesting and you can, for example, take a look at your area in the 19th and 20th centuries as well as look at the present day OpenStreetMap version.

To sum up, lots to look and and worth a look.

London Datastore

I've posted stuff before (e.g. economic deprivation, population growth) about London so this post is about data for London which is available via the London Datastore. It is a website run by the Greater London Authority which you can go to if you want data for London (obviously).



What kind of data is available? Well, you could look at the GLA Budget, average house prices, or even data on abandoned vehicles. You could also take a look at the A-Z list and/or search the archive. Hours of fun.

The data are available at various spatial scales, some below Borough level and some not. Finally, here's a map of Cannabis Possession Incidents recorded by the British Transport Police for a two-year period up to July 2010. This just covers incidents on trains, tracks, stations, etc. Click the map to see full size. The highest value is in Westminster.

Ordnance Survey Data - A Revolution?

Following the news some time ago that most Ordnance Survey data (if not all - still waiting for details) is to be made available free of charge for re-use by personal and commercial users, the GI fraternity is abuzz with anticipation. I am also pretty excited about this development since it will make working with geographic data and using GIS in the United Kingdom much easier. It should also increase the number of users, as SalfordGIS point out.

The current understanding is that from April 1, 2010 several OS datasets (e.g. 1:50,000 colour raster) will be made freely available. What are the implications of this? Just some thoughts...

• Anyone will be able to download and print original OS maps, free of charge
• Anyone will be able to use OS data to derive new maps - previously this was forbidden
• GIS users will use data in new and innovative ways
• GIS consultants will increase in number
• In general, many good things will happen and geographic knowledge will increase
  

Much more will be written on this in the media in the coming weeks and there is no doubt it is an exciting time if you are in any way involved in the world of geographic information in the United Kingdom.

The future is full of possibilities!

Google Motion Charts

This post is about Google's motion charts and how useful they are for displaying data in a dynamic format. In this example, I've taken data for four cities in the north of England, and compared change over time from 2000 to 2008. I've used the following indicators:

• Total population - from ONS mid-year population estimates
• Working age population - from ONS mid-year population estimates
• VAT Stock at end of year - from NOMIS, a measure of business vitality (only got up to 2007 for this)
• Claimant Count - from NOMIS, those claiming unemployment benefit
• House prices - median house price for each area, from CLG Live Tables

You can compare any data against any data by clicking below and beside each axis. You can size the bubbles according to any variable and choose to turn any area's data on or off. They're really easy to use. Try experimenting with the example below, or the larger version I've linked to (note that you can click the tabs to the top right of the chart for a different data view). Pressing the 'play' button in the bottom left of the chart will start the animation.



All you need to make this is a some data and a free Google account. And a bit of time and patience. Once you get how it works it's really easy, and very effective. This is just an example using some English data.

For a REALLY impressive demonstration, see gapminder.

For a larger version of the chart shown below, I've put together a separate page.

Deprivation and Spatial Context - The NNIMD

Time for a short post on some of my more recent work. A lot of what I do deals with areas, deprivation and trying to understand the links between the two. Since I live in Liverpool and work in Manchester, I've got a good idea about how some of the most deprived (according to the Index of Multiple Deprivation) areas look, where they are and how spatially clustered they are. I also think that the way we identify which areas ought to be the focus of policy could do with some refinement.

So, here's what I've done. I took the 2004 IMD data and created what I like to call the NNIMD. That is, the Nearest Neighbour Index of Multiple Deprivation. I've written a paper on this for a journal (to appear around October 2009) but I decided in the meantime to update the work for the 2007 IMD and post on it here. The NNIMD takes the IMD scores for all neighbours of an area and averages them to give what you might call a 'spatial context' or 'neighbourhood' deprivation value for each lower layer super output area.

This process is repeated for the entire country (England) so that for each of the 32,482 lower layer super output areas we have a score which provides intelligence on the kind of neighbourhood it sits in, at least in terms of the IMD deprivation score. Yes, it's not perfect and yes there are issues with the definitions of 'neighbourhood' that we have to use, but it's a good start on the way to understanding local spatial context and the role this might play in neighbourhood outcomes. I plan to continue this work in the future, but for now here's a couple of graphics showing how Liverpool looks using the 2007 IMD (first map) and the 2007 NNIMD (second map). [Note: the colour spectrum goes from blue (least deprived) to red (most deprived).]