Can Google search data predict an election victory?

Today seems like a good day to write a little blog post on what search data can and cannot tell us. Why? Because of the story below, which has been on the front page of the MailOnline for much of the day. This is just the way news works, but I thought it would be useful to give a bit more information here. The story behind the story goes something like this...

Simon Rogers, datajournalist and Data Editor at Google in San Francisco, got in touch some time in April to ask if I could help him map party leader search patterns by constituency. I'd been doing a lot of work with search data for housing markets anyway so this seemed like an interesting idea. We took search data for all points that were geocoded (there were about 5,000 total across the UK) and then produced a constituency version for all 650 seats. The final constituency results matched very closely the proportions for the individual places. The data are for the previous 12 months. 

MailOnline front page 07/05/15

The big question is what this all means. Do I think, as the MailOnline suggest, that 'Google Search tips Cameron to win election'? No. Do I think it disproves it? No. Do I think the large volume of search for Nigel Farage indicates his level of popularity across the country? Again, no. However, it could indicate that people are more likely to show interest in UKIP in an environment when nobody else is watching or listening. But we don't know. Does this prove that Miliband will come third? Definitely not. The map merely indicates who was the most searched for party leader in each constituency. The intent and sentiments of individual users are not known. In my own research in housing market analysis I've tackled this by doing interviews with website users but since this data is from Google they could of course add in other terms which people might combine with party leader names (some more favourable than others!). 

Kate Newton from Bing also got in touch to say they had worked on something similar (though much more sophisticated) in relation to the Scottish Independence referendum last year. More widely, there is a body of emerging research (including my own) which looks at search patterns and subsequent activity - mostly in the field of economics. The results suggest that search can be analysed meaningfully to predict future activity. But that's not what the party leader piece was about - not from my perspective. Thankfully, other media outlets were more measured in their analysis - such as BuzzFeed UK, The Scotsman,  and The Telegraph. 

MailOnline story 07/05/15

What's most interesting to me? Well, I'm most interested to see how the search patterns relate to outcomes in key marginals. I suspect there will not be much of a pattern but if there is it will be interesting to attempt to take this little piece of work further - perhaps for the US 2016 election. Other than that, this is an interesting stocking filler on a day when the papers and TV crews are forbidden from reporting anything really substantial until the polls close at 10am. In the meantime, my favourite snippets from the search map...

David Cameron is the most searched for leader in his own constituency, but he's surrounded by a sea of purple, plus a blob of red and orange.

Witney - David Cameron's constituency

Perhaps a little predictably, there is a lot of search for UKIP in Kent but - strangely - it appears that in the constituency where Nigel Farage is standing (Thanet South) the party leader most searched for is Ed Miliband.

UKIP - lots of search in Kent, but not to much in Thanet South

The search results produce some interesting results. The image below is a good example. Natalie Bennett (Green Party) is the most searched for leader in Durham North West and next door in Durham North the leader (by search over the past 12 months) is Leanne Wood (Plaid Cymru). I suspect this was down to a localised spike in interest after the leaders' debates.

Durham - Green and Welsh Nationalist stronghold?

Other interesting nuggets to emerge were the way in which geographical patterns sometimes reflected the opposite of what you'd expect. The most obvious example was where Nicola Sturgeon (SNP and not standing in this election) was the most searched for party leader in several English constituencies, such as Chesterfield (below). Her excellent performance in the leaders' debates probably led to a spike in interest. Perhaps the SNP ought to consider putting up candidates in England too.

The SNP take Chesterfield? Not so fast.

A kind of similar situation to the SNP/Chesterfield example can be seen in the final image below, where Nigel Farage is the most searched for leader in Aberdeen North. This Scottish constituency has no UKIP candidate and, even if it did, they would be a long way away from the top party.

What's my prediction for the outcome of the election? The only prediction I'll make is that the results will look nothing like this map!

Some tips for charts in Fusion Tables info windows

I've recently been working with some mortgage lending data from banks in Great Britain to produce a new mortgage lending maps site. Once again, I've used Google Fusion Tables to map the data because it's relatively quick and easy - further information can be found here. What's not so easy is getting the info windows to do exactly what you want, particularly when you want to include charts of the type shown below. In this post, I explain a little more about how you can get the info window to display such a chart, what can go wrong when trying to do so and what the code underlying code looks like.

The chart shown above is what you'll see when you click on any polygon in my mortgage lending map website. It takes the data from the underlying Fusion Table and provides a unique info window and chart for each postcode sector - in the case of the above it is for the Cardiff postcode sector of CF24 4, which as you can see had more than £117 million of outstanding mortgage debt owed by 839 households at the end of June 2013. The default info windows created in Fusion Table maps simply contain a number of default data records from columns from the underlying table. With this data, I was keen to show a visual comparison of lending mix in each area, and I wanted to do this using a horizontal bar chart. I'd done info window charts before (e.g. in my Deprivation in Scotland website) but these were line graphs showing change over time.

There is some general help on putting charts in info windows from Google, and this is a very good place to start, and you can also find a lot of explanation for what the different bits of chart code mean in Google's Charts Gallery, or in the Chart Feature List, but to help anyone who might be trying something similar to what I've produced, I thought I would provide an annotated code example. I also want to provide some general troubleshooting advice. Here's what the code looks like inside the Fusion Tables interface:

And here is a Word document with comments added explaining what each bit of code does. You'll notice that it's a bit messy but it produces a very nice looking chart. One thing that I noticed about all this is that if you want the numerical data to appear in the info window with comma separators - as above, for the £117 million figure - then it appears to disappear from the chart. This is what happened to me anyway. My advice would be to keep the number format for your data as 'none' in the Fusion Table field options. My other advice would be to remember to save a good version of your code but most of all to experiment with different things and then share the results.

I hope someone finds this useful! I know I would have before I got into this. 

SIMD2012 - An Interactive Website

With the release of the latest version of the Scottish Index of Multiple Deprivation I thought I would take the time to put together an interactive mapping website so that people who are interested in exploring spatial patterns of deprivation could easily interact with the data. The official Scottish Government interactive mapping site has some nice features but I find it a bit cumbersome and the map interface is too small for my liking so that's why I've produced my own version, based on Google Fusion Tables.

Putting this together has prompted me to develop some additional mapping tools using Fusion Tables code and these can be accessed via the 'Tools' tab on the new website. The 'search and zoom' allows you to enter a place you want to look for and when you hit 'Search' the map immediately pans and zooms to that location.   The other tool I've created simply lets you turn the SIMD map layer on and off, which is quite a useful feature.

I've just looked at the relative ranks of places within Scotland in this site. For details of absolute change you can see the employment and income domain data available from the new SIMD 2012 website.

Unemployment in Europe (via Google)

Despite recent headlines about data capture, Google remains an excellent source of (or gateway to) information on socio-demographic data. For example, if you type in 'population' followed by a country name, such as 'mexico' then this is what you'll get...

If you do this with any country you'll get the latest results plus a little graph which you can then click on and explore further. Similarly, if you type in 'eu unemployment' you will see a little chart showing EU unemployment - currently 10.2% for March 2012 - and how it has changed over time. If you click on the small chart you'll then see data for Europe and be able to add in data for other EU nations by clicking the boxes to the left. You can even embed this in a web page, as you can see below...

Apart from being convenient and accurate, this is also a very useful analytical tool when you need quick comparisons, like in the example below where I've compared Spain, the EU, Germany and Austria. As you can see the time-series data does not always extend as far back as we'd like but it is a great way to get your head round what is happening in different places without much effort at all. You'll notice in the embedded graphs that if you hover over a line it should tell you the data value for that point.

I've now changed the criteria in the chart so that it only includes unemployment for those aged less than 25 - and I've added in the UK too. This makes pretty grim reading for the EU, and Spain in particular...

This method also works for lots of other kinds of data. For example, if you type in 'us gdp' you'll see the data for the US but also have the option to add in lots of other comparators. One of the most interesting comparisons is looking at GDP over time, as you can see below.

I'm going to a conference in China at the end of June, hence my interest in national comparisons. This kind of thing has of course been covered extensively by Hans Rosling, but not many people know that it is fully integrated into Google's basic functionality.

Mapping Methods

I've done a lot of mapping on this blog in recent months. Much of this has been about deprivation and my attempts to make more widely available maps on deprivation for different parts of the UK. For this, I've often used Google's Fusion Tables. The most recent work I've done with this data using Fusion Tables is to update the Welsh Deprivation work to include the most recent release of the Welsh Index of Multiple Deprivation from 2011. There's a screenshot below which links to a full page Google map. If you click on an area the pop-up will tell you all about it in relation to the WIMD data. This post isn't about the data but the most deprived area is in West Rhyl and the least deprived in Cardiff (Llandaff area).

But this post is about methods, so more on that before I go... The steps below relate to any kind of data I've mapped using Fusion Tables (warning: technical content!).

1. First of all I usually have to join attribute data to spatial data. I do this in ArcGIS but it works well in MapInfo too. If you're a MapInfo user and want to follow the steps below, use Universal Translator in MapInfo to convert the file to a Shapefile first.

2. I don't like overly detailed boundaries because of the large file sizes and often this exceeds the Fusion Tables file limit. So, I simplify the boundaries. For this, I use mapshaper, a great online tool. You can also use other GIS methods.

3. Then I use something called shpescape. This is a really great tool because it allows you just to zip your shapefile (i.e. the shp, shx, dbf and prj files) and then upload directly to Fusion Tables without having to convert to KML as an intermediate step.

4. Once there, all you need to do is go to Visualize / Map and then go to work customising things. This includes map colours and what appears in the Info Window pop up when you click an area. I've blogged on the Info Window code bit before.

5. If you want others to see it you must make it public. Just click 'Share' in the top right of the Fusion Table screen.

That's about it. The Info Window code bit takes a while to figure out but you can do so much with it.

World Population and Projections

I've been thinking about the future a lot recently because I've been working on a proposal involving scenarios and visions for the future of Europe. So, time to post a couple of things on population change and projections. The first thing is that if you type 'world population' into google, you'll get the following results:

Clicking on the graphic will take you to the google chart where you can turn different countries on and off by ticking the boxes. This makes it easy to compare data on population change from 1960 to 2009 (as below).

The data are from the World Development Indicators of The World Bank. But what if you want to know about the future. Well, there are lots of projections for different countries, but I don't think there are any that go as far as the 2004 report from the United Nations, entitled World Population 2300. Yes, the year 2300.

This is a 254 page report by the Economic and Social Affairs division and they firsst consider projections to 2050 and discuss assumptions and long-range possibilities. Part 2 looks at different scenarios and different areas of the world (e.g. Oceania). Other parts look at country rankings, population density and ageing. The world population projection for 2300 is projected to be about 9 billion. This has already been covered in a Worldmapper map, so no map here. That's all for now...