Temporal Change: Census Data¶

In the Active Remote Sensing Notebook, we looked at change over time with raster data, using SAR satellite imagery collected months and years apart to map floods in Athlone.

Change over time is not limited to raster data - and census data is an excellent example of that. Population estimates for Ireland have been made since the population was counted at 1,100,000 in 1672, and the first full census of Ireland was taken in 1821. From 1841 to 1911, a census was undertaken every decade. Following the War of Independence, the first census of the State as it is today was taken in 1926, continuing at 10-year intervals until the interval was reduced to 5 years from 1951 onwards. The most recent census was delayed by a year due to the COVID pandemic, with the data published last year.

The census doesn’t just count the number of people. If you were living in Ireland on Census night - 03 April 2022 - you should have filled in the form, and so you might have seen the range of questions. In addition to the number of people present in a household on census night, there were questions about the age, sex, marital status, place of birth, ethnicity, religion, languages spoken, health, commuting and travel, education, and employment for each person. There were also questions about the household - the age of the building, ownership or rental, provision of energy and water supply, and more.

Most importantly, the census forms are specific for each address. This means that we don’t just know who, but also where - and we can put that on a map.

And because we have census data going back decades, we can also map changes in the data over time.

We’ll need pandas, geopandas, numpy, and matplotlib for this Notebook.

1. Mapping change over time in vector data¶

Census data is vector data, and this will apply to any vector data. In order to illustrate change over time in vector data, you need three things:

1. Vector geometry features

2. Data for each vector geometry feature at an earlier data/time

3. Data for each vector geometry feature at a later data/time

Let’s demonstrate using some census data for Limerick from 2016 and 2022. To do this, I’m going to use a couple of files which are identical to the full census data, but for which I’ve isolated only the data for Limerick City and some of the surrounding area (we could use the full data, but using the selected area will just make things a bit quicker since we won’t be downloading or processing the entire dataset).

First, the 2022 map of vector geometry features:

And actually, if we look at the data, we can see it has the 2016 and 2022 GUID codes in different columns, so we won’t even need a 2016 version map.

Now the 2022 data:

and the 2016 data:

That’s the three elements we need, but they’re in three separate files. We can’t use that directly. Fortunately, we can use the GUID codes to link the data to the vector geometry.

To keep things simple, we’ll just look at raw population numbers: so rather than just merging the full datasets, we’ll extract only the GUID and population for each row from the data tables.

We could show that with two maps side by side:

The very dark area showing a population close to 800 is the hospital, by the way.

So, some changes are pretty clear, but it’s still a bit awkward looking side by side. Why not try and show it on a single map? We have a single vector geometry geodataframe, so we can just add both population data columns into that instead of making it a new geodataframe for each year.

Scroll right, and you’ll see both 2022 and 2016 population data columns.

However, if we want a single map, we need that data combined in a single column. So, we make one by subtracting the two columns we just added.

Scroll right to see the population change column we just made. Now we can plot this:

We can make this look even better if we redo it using the same trick as for the DEM terrain map in Notebook 9, making a colourmap centered at zero:

Now that looks a lot better! We can see the changes a lot more clearly here.

And that’s the simple version of how to show change in census data over time.

2. Census Geographic Areas¶

Yeah, there’s a ‘BUT’ coming.

Before we start really trying to analyse change in the data, let’s just take a moment to think through exactly what we just did above, what change we’re showing. In order to really think about this, we need to start by considering what’s actually in the data.

The data we used above is just parts of the full census dataset, so let’s have a closer look at them, starting with the data table.

This is some of the census data for 2022. The columns to the right contain the actual data: by reference to the glossary, we can see what the column names refer to, with the visible columns T1_1AGE0M being a count of the number of male children 0 years old, T1_1AGE1M being a count of the number of male children 1 year old, and so on. We used T1_1AGETT above because I already know it’s the total population of the area. But what about the other columns?

The first three columns are the geographic identifiers, GUID standing for Geographic Unique IDentifier. Each row then refers to a different geographic area identified by the ID in this column, and of course we used this to link the data to the vector geometry shapes. But how are these geographic areas defined?

These particular areas we just used above are referred to as ‘small areas’, and they are defined specifically for the census. The forms for the census are tied to individual addresses, but it would be quite intrusive to publish the individual data - which includes answers to how much people earn, when they travel to work, and other sensitive information. So, the data has to be collated to be shared, in order to protect everyone’s privacy. Collating the data over large areas can obscure useful information, though - for example, if you want to see whether people are using a particular bus service. The small areas are essentially the compromise: publishing the data for quite small geographic areas, but not so small that individual people’s answers could be identified.

The small areas are defined such that each should normally have in the region of 100-200 people, and they try to keep the boundaries of the small areas meaningful where possible - following features like roads, housing estates, and similar.

However, the above is just the table of data - it’s doesn’t have the actual polygon shapes for the small areas. Of course, we joined this data table (or part of it, anyway) to the vector geometry above. Let’s look again at that vector geometry geodataframe:

Here you can see two GUID values for each small area - SA_GUID_2016 and SA_GUID_2022. We used both of those above, to join the data for 2016 and 2022, but we didn’t really talk about it. If you look at the next set of columns, you’ll see SA_PUB2011, SA_PUB2016, and SA_PUB2022: that’s reference numbers for small areas for the past 3 censuses (censii?).

Why do the small areas need different reference numbers for each census?

Because the small areas change for each census.

The small areas need to change for the same reason that the census is run every 5 years. Not all small areas change every time, and they try to keep the changes to a minimum; but as the population grows, the number of people in some of the small areas will increase, urban growth will see new housing estates built in previously rural areas, and so on.

This does mean you need to be careful, because as we said at the top, in order to look at change in an area over time, you need three things:

1. Vector geometry features

2. Data for each vector geometry feature at an earlier data/time

3. Data for each vector geometry feature at a later data/time

And here’s the real ‘BUT’: if the geographic area itself is changing, then the change in the information (such as population) is not meaningful.

Think about what we mean by change: to look at how anything changes, we need some information which is changing, and some which is staying the same. A bus moving faster means it is travelling a larger distance (that’s the information which is changing) in the same duration of time (that’s the information which is staying the same). We can’t say the bus is faster if we’re just comparing distances travelled over different durations of time - or if it’s a different bus. We can’t say a child is getting taller if we measure a child’s height one year, and a different child’s height the next year. Has to be the same child, obviously. Similarly, for census areas, if the population grows and the area changes, what’s staying the same?

There’s two points to saying this. First, that you have to be careful when working with census data that you’re comparing like with like; since some of the small areas change each time, that needs to be accounted for. The fact that the small areas map data contains GUIDs for 2022 and 2016 means that this is generally possible for the most recent censuses, but is awkward in cases where areas have been split or reconfigured because it’s not possible to know how areas have been split or reconfigured. Secondly, it means that the further back you go, the less data you will usefully have, because of the greater changes to the areas.

In the example above, I specifically chose Limerick because there’s actually only very minor tweaks to the small areas between 2016 and 2022 - or at least, only very minor changes within the area shown. I deleted all the areas with significant changes. So it’s definitely an oversimplified example, and you won’t be able to do that everywhere.

Let’s look at an example with more change.

3. Population Change¶

Let’s look at Carlow from 2016 to 2022. I’ve already merged the population data with the vector polygons for the small areas, using the 2016 and 2022 small area polygons separately.

We can see that the population has increased overall, and we can see some particular areas where the population has increased. But, we can also see that the small areas themselves have changed. We can highlight this by plotting the 2022 small area boundaries on the 2016 map, and the 2016 small area bouundaries on the 2022 map:

We can try to do what we did for Limerick, and put both population columns in the 2022 geometry geodataframe, using the SA_GUID_2016 and SA_GUID_2022 to link the 2016 and 2022 data respectively. In fact we already have the 2022 population data in that file, so we can add the 206 data easily in the same way we did for Limerick earlier. So first extracting the 2016 GUID and population numbers from the 2016 data, and then using the SA_GUID_2016 to join this with the 2022 geometry.

Now I can join this to the 2022 data:

Scroll to the right - you should see columns for both Population_2016 and Population_2022. Now we calculate a new column showing the change by simply subtracting these two:

Let’s plot this:

That looks nice - but remember that because some of the areas have changed, this isn’t actually showing change in population for each area. Some areas have ‘lost’ population because they’ve been divided, or had parts shifted to ther small areas, not because of actual increases or decreases in population. Only areas which have actually stayed the same will be showing meaningful changes in the actual number of people within that area:

4. Alternative Geographic Levels¶

There’s not really any easy way to account for the changes to the small areas themselves, unless the CSO actually republish the 2016 census data using the 2022 small areas. That’s unlikely, unfortunately, because they have too much work to do already on the new data - not all the 2022 results are released yet, and they’re already developing the next census for 2026.

However, if we use larger geographic areas, there would be fewer changes. While not giving us the same spatial resolution, it’s often enough for what we need. The census data is also published at the level of:

• Electoral Divisions (ED)

• Local Electoral Area (LEA)

• Administrative Counties

• NUTS3 Regions

• Provinces

Electoral Divisions were historically areas which elected local councillors, but with the change to multi-seat proportional representation areas, they’re now generally of low significance - and so aren’t changed very often. There were 3409 EDs in 2016, and 3420 in 2022 - an increase of 11, I think all due to the expansion of Cork City - although there were a small number of other tweaks as well. This makes them useful for comparing census data over time. They do have two drawbacks though: first, since they’re historic areas, and so their populations vary wildly, from fewer than 10 people to multiple thousands; and second, for the same reason, their the areas covered by different EDs also vary wildly, from a few city blocks to areas larger than entire cities.

Local Electoral Areas are the new districts from which local councillors are elected. These are revised from time to time based on population changes, so they’re less useful for comparison - but of course it’s useful to look at them in terms of them being areas represented by councillors, and so subject to political attention.

Administrative Counties are the areas covered by local Councils. These are not changed regularly, although it’s not all that long since Limerick City Council and Limerick County Council were merged into Limerick City and County Council. There’s some large-scale change comparisons possible, but they’re too large for a lot of possible change over time analyses.

NUTS3 regions are European Union data-reporting regions. They’re useful for international comparisons, but not really within Ireland.

Provinces are historic, and too large for most meaningful change analyses - I’m really not sure why the CSO still reports the data at this level.

Raw population numbers are also published in 1km grid squares, although this doesn’t come with the other census data.

The population change looks smaller at ED level, yes?

Unfortunately, plotting a direct comparison on a single map is made more awkward by the CSO using different map polygons with different GUIDs, and the only shared attribute is called “ED_ID” in 2016, with leading zeros, and “ED_ID_STR” in 2022 without leading zeros. Meaning I had to manually edit these to match them up.

Anyway, I did it, so here:

Honestly, cleaning the data to make this took me about 5 hours. Do not recommend unless important. Which just serves to reinforce the point that you’ll probably spend most of your mapping work time cleaning data, unless you’re very lucky.

So, in theory, census data is great to show over time - but be careful, because it’s not always as straightforward as you might think.

5. Beyond Population Data¶

Of course, for areas which don’t change boundaries, you can do a lot more than just look at the raw population numbers.

The next steps here would be to add some bus routes, stop locations, stop catchment areas, cycle lanes etc. Hopefully you can see the potential - just be wary of the data cleaning needed to make it work, if it’s possible.

Summary¶

The key point here is that if you want to show change over time in vector data, at the core it’s no different from showing any other vector data. You need two things:

1. vector geometry

2. data in an attribute column

just like any other vector map.

The only difference is in where the attribute data comes from. Normally, it will involve some simple maths - subtracting data for the earlier date from the data for the later date.

The complicated part, as in the example above, is in ensuring the data for both dates refers to exactly the same geometry shapes.

There’s simple examples where this is less of an issue - for example if you want to illustrate data for countries, it’s mostly straightforward, although you do need to watch out for examples like East and West Germany.

But for census data (and of course the same might apply to other examples), this isn’t straightforward, and often requires a lot of checking and data cleaning to get the data into a usable format.