Natural Earth for GIS data

People often ask me where they can get GIS data to use for projects, analysis and general mapping. In the UK we now have OS OpenData, which is very nice and very detailed. There's also a new GIS portal from the Office for National Statistics - built using the Geoportal Server. Other GIS datasets are available, from organisations like Natural England, but in this post I thought I'd highlight the excellent - and totally free - Natural Earth site which is very well known in the geodata community but perhaps not more widely. It really is absolutely fantastic.

A little bit of technical information....

• Natural Earth Vector comes in ESRI shapefile format, the de facto standard for vector geodata. Character encoding is Windows-1252.
• Natural Earth Raster comes in TIFF format with a TFW world file.
• All Natural Earth data use the Geographic coordinate system (projection), WGS84 datum +proj=longlat +ellps=WGS84 +datum=WGS84 +no_defs

Whether you're looking for data to make a general world map or a more detailed local area map you will find what you are looking for here. It's not as detailed as OS data for Great Britain but then many GIS users don't need that level of accuracy. Another screenshot below showing the various download options...

Finally, taking my inspiration from the worst website in the world, and Ken Field's blog - I've made an absolutely awful map using some of this data*. Can anyone do a worse one?

*I actually did this as part of my work in developing some new GIS modules. I tried to pack in as much bad practice as possible just as an extreme example of what not to do.

Mapping flows in ArcGIS

This short post is about the process of flow mapping in ArcGIS and not really about the end results - though the maps are quite interesting. I've done quite a bit of flow mapping in the past and am now getting ready to work on the next set of Census flow data in the UK (which should be out in November) so I've been experimenting with some tools. I've written about this in the past in papers in Computers, Environment and Urban Systems and also Environment and Planning B but those papers are a bit long-winded! Other people have produced beautiful flight path maps so I thought I'd experiment with the same data using the relatively new ArcGIS XY to Line tool in version 10.0 (it can be found in ArcToolbox - Data Management Tools - Features - XY to Line at the bottom of the list of tools). For more on other methods and previous iterations of this kind of thing take a look at the work of Nathan Yau, Michael Markieta or James Cheshire.

For anyone wanting to map flows in ArcGIS, Michael Markieta's tutorial is probably a good place to start but be prepared for things to go awry in ArcGIS... When I mapped the 59,000 or so flight paths in the map above using XY to Line and one single dbf file (or csv, etc. - it makes no difference) the resulting shapefile only contained 16,066 rows. This happened every time I tried it and a couple of times my shapefile had only 73 rows. Another time it had ~14,000. That's why Markieta recommends splitting the file up - although I just cut it up into chunks of 16,000. Interestingly, I ran into exactly the same problem with my CEUS paper a few years back using Glennon's flow data model tool - though the limit was about 32,000 before it cut off. 

Another very annoying feature of XY to Line (for me at least) is that when you choose the Great Circle option under 'Line Type' it takes much longer to compute and the resultant shapefile is enormous. The shapefile for flight paths in the above map is about 10MB whereas the great circle version was over 450MB for one 16,000 chunk alone. Not sure if anyone else has run into this but it doesn't seem like a very efficient way of doing things! [Edit - as @baeing has reminded me, it's because shapefiles don't support curves - though geodatabases do.]

Once I had my complete shapefile I moved to QGIS, added in a world layer from Natural Earth and then experimented a little with symbology. I also experimented with different styles and projections to produce some of the maps below. That's all for now - I just hope ESRI are able to improve upon the current version of XY to Line because when it does work it is really fast (on my machine at least) and straightforward.

Very similar to above, minus text

Short haul, different projection

Short haul, different projection, borders

Slightly different symbology

And, yes, I know that flights from Australia or New Zealand typically go over the Pacific rather than the long way round! I'm just showing the XY to Line outputs as they are in this post.

Employee Growth in London, 2001 to 2012

The Office for National Statistics has released a new dataset on the number of employees across London's 983 MSOAs. The data are sourced from the Inter-Departmental Business Register (IDBR) and they reveal some interesting trends. Naturally, I had to do a 3D map of this, so take a look at the image below for the obvious growth points...

Massive absolute growth in the City of London and Canary Wharf - and some other central MSOAs in Camden, Southwark and Westminster, but also massive growth in employment in Uxbridge.

Click on the image to enlarge

The number of employees in the City of London increased by 36%, compared to 267% in Canary Wharf and 170% in an Uxbridge MSOA. By contrast, one part of Islington had 78,600 employees in 2001 but only 57,000 in 2012 - a drop of 27%. This area of Islington is immediately north of the City of London and includes Clerkenwell and Finsbury.

If you're interested in this kind of thing it's definitely worth looking at the original dataset.

Stratospheric London

I've recently been looking at the Land Registry's price paid data for England and Wales. This is now open data and the Land Registry recently published all data going back to 2009 and by November 2013 they have promised to release all price paid data going back to 1995. I've been looking at the period from January to May this year to see - at a very basic level - what is happening in terms of the total volume and value of sales across England and Wales. As expected, this simply tells us that the London property market is 'stratospheric', as the Evening Standard recently reported. A few summary stats and charts before I go on holiday...

• Total value of sales in England and Wales between January and May 2013: £51 billion.
• Total value of sales in Greater London: £14 billion (28% of the total)
• Total value of sales in Westminster: £1.6 billion.

Explore the data and charts for yourself in the spreadsheet I put together here. The original data comes from the Land Registry's price paid data. I've used the original source files to produce some summary stats by Local Authority, City/Town and County. If you want to know more about the data, make sure you read the Land Registry's FAQs. 

Data produced by Land Registry © Crown copyright 2013.

This data covers the transactions received at Land Registry in the period 1 January 2013 to 31 May 2013. © Crown copyright 2013.

The London Problem Revisited

I've been thinking about the matter of London and the kind of 'two-speed Britain' stories which have been in the press recently. The idea is that London's continued growth and prosperity - in contrast to the rest of the UK - is a problem. The BBC covered this via Stephanie Flanders in March and the Guardian did a piece on it in May this year. The Economist even did a special supplement on London last year, entitled 'On a High' in which they covered a range of issues, not just economic growth - e.g. a map and some of my work on deprivation on p. 7 of the pdf. These kinds of stories always generate a lot of debate (see the comments on the Guardian piece for an example) of the 'London vs. the rest' variety but I suppose the thing that sticks in my mind is that none of this is really new. The precise nature of London's recent resurgence is perhaps unique but it's hardly a new problem. I was reminded of this last week at the Centre for Urban Policy Studies 30th Anniversary conference in Manchester when Sir Brian Briscoe quoted from the Barlow Report of 1940 (photo below - yes, I actually have a real copy to hand!). 

This in turn reminded me of the 1985 edition of the Planner magazine that I've had on my desk for some time. Why do I have this? Because it's a special issue to commemorate the life of former Chief Planner and University of Sheffield Professor JR James, who featured on the front cover of the April 1985 edition, five years after his untimely death in September 1980. In this edition, leading planners of the time gave a series of lectures on seminal reports - Peter Hall on The Barlow Report, Gerald Wibberley on The Scott Report and HR Parker on The Uthwatt Report.

Peter Hall's reading  of Barlow is fascinating for many reasons but I found it interesting that he chose to pick out the exact same quote that Sir Brian Briscoe did last week: "the continued drift of population to London and the Home Counties constitutes a social, economic and strategical problem which demands immediate attention" (p. 202 - see below for copy of the original, in paragraph 5).

These words were written in very different times (e.g. it was published shortly after the outbreak of WW2 and in fact some of the content had to be changed to reflect this) but the over-arching message of a dominant London and a lagging rest of Britain resonates today. The policy solutions proposed in 1940, however, are rather different to those suggested today. For example, there was a strong emphasis on regionalism in the Barlow Report - unlike today. Further, there was a real concern about what could be done to address this nationally problematic uneven development (well, at least in theory) - e.g. Part IV from p. 185 is entitled 'Remedies ... And to Report what Remedial Measures, if any, should be taken in the National Interest'. So, this is not just an historical curiosity. On p. 197 we have more on 'methods of decentralisation or dispersal', which many today would like to see.

As for the man himself, Sir Anderson Montague-Barlow (image at bottom of page), I'm not sure what he'd make of the current rhetoric of a 'two-speed' UK but I'm sure he'd recognise that it's nothing new. The question that remains in my mind is whether this London boom will continue indefinitely and what the consequences of that might be. Some final thoughts for now...

1. Lost in much of this two-speed UK talk is the fact that inequality and poverty in London are quite extreme. That's what I was getting at in my very small contribution to the Economist supplement last year and in another Guardian piece. This is something that Ben Hennig has been looking at in more detail in recent times - see here for more.

2. Much less has been said about why this London/rest issue might be a problem and - more importantly - what might be done about it. Certainly, I don't think we'll see a Barlow-style Royal Commission about this any time soon.

3. What about the contribution of the rest of the UK to London's growth - e.g. in sending graduates, commuters, funding for infrastructure, and public transport - to name just four factors. The question here might be to what extent is London's growth down to the fact that it has a Barlow-esque gravity? Easy to answer but harder to quantify.

4. Are we witnessing a 'London supernova'? Will this bright shining star gradually fade? Fancy rhetoric, but worth thinking about, given the cyclical nature of boom and bust. See p. 52-54 of this RTPI report for more.

5. And, finally, we ought to look at old books and magazines more! 

HS2 geodata - for download

Yesterday I wrote a short post on the Guardian's Datablog about my difficulties getting hold of the route data for the proposed routes for the new high speed rail lines in England. Coincidentally (or maybe not) HS2 responded to my request at almost exactly the same time as the piece appeared online. Anyway, sometimes it does take time for public bodies to respond to requests so my real question was why the shapefiles were not available for download, given that they are available under the Open Government Licence. I have a few ideas about why this must be but it would be good to have some information from this on HS2, though maybe they're too busy with other things! Clarity on this issue might, however, reduce the likelihood of data conspiracy theories and enhance transparency.

Anyway, enough about that. So that other people don't have the same wait as me to get hold of the GIS data I've made them available here via the link below. A few important points to bear in mind...

1. The Phase 1 (London to West Midlands) route is the 'post-consultation' route from January 2012.

2. The Phase 2 (Leeds and Manchester) routes are the 'initial preferred routes' from January 2013.

3. There is an interactive map of the Phase 1 route on the HS2 web pages, which is quite useful.

4. Users of the data need to remember to acknowledge the source.

5. It's not my data - I'm just making it available.

6. You can also get these files from Barry Cornelius, but not - as yet - from data.gov.uk

7. The route data available for download here doesn't necessarily reflect the precise location of where the train lines will be built - particularly for Phase 2. 

HS2 shapefiles, as of 2 May 2013

Population 'explosion' in English city centres

A paper I wrote about English urban policy and the 'return' to the city is now out in Cities, so I thought I'd blog about it. It's just something I wrote after the first release of small area data from the 2011 Census and the results are not entirely surprising to those with a knowledge of these things but the scale of change over the decade from 2001 to 2011 was pretty big, particularly in the case of Manchester, which I've written about previously. I also reflect upon wider international issues associated with 'reurbanization' (with a z because it's a US journal!) so I think that although the focus is on England it should have wider resonance.

I discuss this growth in the context of New Labour's urban policies, and particularly those which emerged from the Urban White Paper of 2000. Although some of the aims were achieved (e.g. making city centres look nicer, with better design), I conclude that the changes have been mostly superficial and that perpetually high levels of inner city deprivation in cities which were the main foci for urban policy during this period does not represent a very positive legacy, despite the 'success' of getting people to move back to the city. The fact that this picked up immediately by @urbandata suggests that these themes are also relevant in the United States, and beyond. It's a short paper but I hope to follow up on it at some point* with more detailed data on tenure, etc. to dig a little deeper.

*This is normally code for 'it will never happen'!