Podcast: Chaos in employment statistics

Participants

Dr Hannah Randolph – Fellow at the Fraser of Allander Institute

Allison Catalano – Fellow at the Fraser of Allander Institute and the Scottish Health Equity Research Unit (SHERU)

Time stamps

(00:06) Introduction

(01:20) What is the LFS and what are the issues it’s been having?

(6:57) Alternative data sources for understanding the labour market

(18:23) Misunderstandings around what’s happened in the post-pandemic labour market

(19:39) ONS changes to improve the quality of the LFS

(23:40) How poor employment statistics may impact local decision-making

(27:49) Information about FAI 50th anniversary conference

Transcript

00:00:06 Hannah Randolph

Hello and welcome back to our latest Fraser of Allander podcast. I’m Hannah Randolph, an economist here at the institute, and today I’m joined by Allison Catalano, who is also an economist here at the FAI, who is also working on the Scottish Health Equity Research Unit, or SHERU. Thank you for joining me today. Would you like to introduce your area of research for people listening?

00:00:22 Allison Catalano

Thank you, Hannah.

Yeah. So like Hannah said, I’m an economist at the FAI and I also work at the Scottish Health Equity Research Unit, so, SHERU.

So a lot of what we do is, you know, we look at what the socioeconomic determinants of health are and health and equality in Scotland specifically. So rather than, you know, being an epidemiologist, rather make someone who honestly really does a lot of work with health, specifically, we look at the things that sort of build into health. So that’ll be things like our socioeconomic circumstances, our household income, jobs, housing and a lot of stuff with poverty and and things like that. So a lot of the work that I do specifically is on employment. So I look at economic inactivity, labour market participation and employability programmes across Scotland.

00:01:20 Hannah Randolph

Both you and I use a lot of survey data for our work. And so today we’re here to talk about the Labour Force Survey specifically and also the annual Population Survey a little bit, which is based on the LFS.

And so both of us have used that before to look at, for instance, trends in employment, economic inactivity, how much of inactivity is attributed to poor health, which is something that’s very important for what you’re doing. But the LFS has been having a lot of problems with quality.

00:01:48 Allison Catalano

Yes. So the the LFS is, I think it’s probably not an overstatement to say it’s like arguably the most important survey administered in the United Kingdom. Yeah, in Great Britain, it’s administered by the ONS. Northern Ireland actually does it different. So when we’re talking about this, the caveat that it is the LFS specific to Great Britain that’s having issues. Northern Ireland is doing totally fine, like no problems there, but the ONS-administered LFS here has been having, you know, falling response rates. Which means that, increasingly, since the pandemic, it’s been less and less and less reliable.

00:02:24 Hannah Randolph

And that’s important because it’s the basis for all our employment statistics.

00:02:28 Allison Catalano

Yeah, it’s the basis for our employment statistics. It’s also the basis for how the Bank of England sets interest rates. It’s the basis for a lot of the, you know, how do we know that we’re we’ve got a good economy. How do we know that things are going in the right direction? We look at employment figures and unemployment figures, and why people are not in work and what they need to do to get into work, at the sub national level and below that. So the Scotland level, like, at this point those statistics are just not really reliable at all. And the Annual Population Survey which essentially takes an average of of the Labour Force Survey over a year is also just not reliable at all at this point, and hasn’t been for a couple of years now.

00:03:15 Hannah Randolph

So if you pay attention to anything related to employment statistics, you’ve probably heard in the news about some of these issues. Like you said, there’s been falling response rates. So it’s really hard to get people to respond to the to the LFS and to other surveys. But this one in particular is really important. And there’s also been some issues with, like, things like culture and leadership within the ONS, right?

00:03:37 Allison Catalano

Yeah, there was an independent review that came out just a couple of months ago, two months ago back in June, which was an independent review that looked into why this is happening, you know, why are these statistics so bad? I should say the way that we know that they’re bad, which we’ll probably talking about a little bit more in a little bit, but the reason we know that they’re bad is because you can look at other sources of data like administrative data from HMRC, tax data, and you can see that in the UK, what we’ve been noticing is from the LFS, employment rates have been going down pretty dramatically or they went down pretty dramatically 2021 to 2022. And what we saw if we looked at tax data is that employment levels had gone up.

And that was a really confusing thing. It’s like, why aren’t these two things aligning? Where are all these sick people coming from? What is going on? And it seemed like, you know, we started to get the sense that things were just not right. And so again, that goes back to 2021, and now we’re in 2025.

We’ve had years of issues that have been very well known. At one point, the statistics were no longer listed as official credited statistics, which in the statistical world is a huge, huge like nightmare scenario.

So this this independent review that came out just a couple months ago found that some of the problems came down to the ONS culture itself. I don’t know if, if toxic is really the right word, but it seemed like there were a lot of management issues, issues with planning, issues with budgeting.

I think the quote literally is “the end results looked to many in ONS closer to equal shares of misery”. So not a great, not a great world for people working in in the civil service around this.

One of the things they also did is they deprioritized funding to core central surveys like the LFS, which again one of the most important, probably the most important survey in the United Kingdom, deprioritized funding to things like that and moved it to experimental statistics, which is a cool thing to do. You know, we should be looking to innovate. We should be looking to drive change but it – this is a really key example of why we can’t do that. Like we can’t devalue these main statistics to drive innovation. We have to find a way to keep these statistics reliable and also invest in innovation in the future because the end result is that now we’ve had inaccurate data since 2021 that we can’t go back and make accurate so forever, we’ll have this black hole when we look at UK economic statistics, that we just cannot know anything about.

00:06:31 Hannah Randolph

And one of the things that’s important for statistics like these is to have continuity as well. And so it’s like, whatever we’re doing, whatever OS is doing to fix these, potentially causes, like, a break in the series.

00:06:46 Allison Catalano

Yeah.

00:06:47 Hannah Randolph

And I know that they’re trying really hard so that that doesn’t happen, but potentially you have something where it’s like now these statistics are not really comparable to the ones from prior to the pandemic or something like that.

00:06:57 Allison Catalano

Yeah, they’re like, they’re not comparable or they’re just not usable. And and the ONS and Scottish Government have come out and say that the very headline statistics are reliable at this point. So that’s going to be employment, unemployment, economic and inactivity. But when you want to drill down to things that are used by policy, like every government and UK is going to use these and they’re going to use these for more things than just employment, unemployment and inactivity. So a lot of work that I do and that Hannah also does is on health and inactivity.

So what are the reasons why people aren’t active? What are the – do these people want work? And at this point you can’t use that data at all. So if you are in a local authority and you want to have an employability programme that works towards people, you know, who are in poor health, you don’t know anything about what that population looks like, like that data just isn’t reliable now. You can go back to 2014 or 2019 and see, you know, how many people in your local authority were inactive due to ill health. How many of those people wanted to work, what sorts of conditions that those people have, and you could feel that those were generally reliable. And and today you just can’t do that.

00:08:14 Hannah Randolph

Well, then, there’s the census from 2021.

And that’s, that has some information about employment and maybe causes of inactivity, but also at this point we’re now three or four years past that. So that could be less –

00:08:28 Allison Catalano

Yeah. So, so 2022 in Scotland, and I think during – at the time of the census, that people had some concerns about that because the response rates to the census weren’t amazing. You know, what you want is, obviously, that it’s universal and we didn’t get it, but it’s still, the vast, overwhelming majority of Scottish people in 2022 responded to the census, and there were a couple things that, you know, for for me and my job were a real concern. And one of them is that if you compare the 2022 census to the 2022 Annual Population Survey, they have a pretty, they have the same rate of economic inactivity, but the reasons are kind of all over the board.

Like the APS found that around 250,000 economically inactive, working aged adults were inactive because of sickness or disability, long term sickness or disability. The census found that it was 200,000, so 50,000 more sick people in the APS, plus temporarily sick or injured. So it’s basically like 75,000 extra sick people in the economy, according to the Annual Population Survey, compared to the census, and at the same time, the Annual Population Survey is undercounting people who took early retirement and students.

Which, like you know, in the past, in the years since, the number of conversations I’ve sat in with, you know, policymakers were like, what are we doing about this, why are so many people in the economy sick? What can we do about this?

Really, looking at the Annual Population Survey, the growth in people who are inactive due to ill health has been really alarming and it seems like that’s just maybe not happening to the extent, anywhere close to the extent that we thought it was. So this has been a huge policy concern for several years that maybe was, you know, should still be a policy concern, it’s still worrying that people can’t work because they’re sick, especially if they want to work and they just can’t do it but it seems like it’s just been dramatically, dramatically overstated.

00:10:30 Hannah Randolph

In terms of alternatives to the LFS and getting an idea of what’s going on, you’ve mentioned a couple of things now. There’s some administrative data and there’s the census, both of which seem to be saying that inactivity is maybe lower than what is represented in the LFS, or at least that fewer people are inactive due to poor health. So can you say a bit more about the alternatives and what they’re showing?

00:10:54 Allison Catalano

Yeah. So if we’re looking at the LFS, looking at the way that the number of employed people in Scotland from the UK are counted, you take the number of people, people who are employees, so payroll employees, the number of people who are self-employed, unpaid family workers also count on the Labour Force Survey and the numbers that are fairly small. I think there’s about 10,000 in the most recent recent count, which may not be totally reliable, but there are some, and then people on a government supported training employment programme.

So those four populations make up employed people according to the Labour Force survey. So if we wanted to take administrative data and copy that, we would need those four datasets. So for employees, we have, there’s something called PAYE data, which is really responsive. We get it in every month. Pro, we have it at the Scotland level, we have it below Scotland level. It is very fast, very responsive, very reliable and it’s tax data. So it’s also counting real – everybody. It’s everybody. Yeah. So it’s not, it’s not counting a small population and then applying a weight to that to kind of get it to be representative. It is basically representative.

The problem with it is that it counts payroll employees and not unique people. So if you have two payroll jobs, you’ll be counted twice. So there’s a little bit of inflation there. I think it’s fairly small. That’s an issue.

00:12:34 Hannah Randolph

It’s what we have.

00:12:36 Allison Catalano

It’s what we have! And other countries get around that and the fact that we don’t is concerning and means that when we have issues like this, like we’re in a really bad spot. So the second thing is self-employed workers, that is not responsive and that is not really available at the Scotland level in any way that’s usable. I think the most recent data is 2022/23.

So you know, if we want to model something without using the Labour Force Survey, we have to use pretty old data and make some pretty big assumptions about what that looks like today. We don’t know how many self-employed people are in Scotland. If we wanted to know how many people were only self-employed and not on payroll data, we don’t have that data for Scotland at all. We have it for the UK, we don’t have it here.

And then there’s unpaid family workers and government supported employees. There is no data for that outside of the Labour Force Survey, so we don’t have any way of gathering that at all. So some pretty difficult, it’s difficult to kind of make an assumption about this, however we did do it. So we – so back in, I think, November, the Resolution Foundation, Adam Corlett from the Resolution Foundation published a paper that did modelling for the UK using mostly administrative data. Some of this data that you cannot get anywhere else from the LFS. And he modelled trends in employment since 2015 and so the way that this is working is you’re taking these four groups of people who are counted as being in employment. Yeah. And then using that to say, OK, and then everyone else in the population is.

00:14:13 Hannah Randolph

So the way that this is working is you’re taking these four groups of people who are counted as being in employment and then using that to say, OK, and then everyone else in the population is not in employment.

00:14:19 Allison Catalano

Yeah. So what you can do is, you can get a pretty reliable figure for employment going back to like, again, we modelled it back to 2015. And so they did this back in November, we did the same thing for Scotland. There are a couple of extra assumptions that you have to make there, but we did the same thing. We got pretty much the same results going back to 2015. And we found that it mirrored LFS data up until 2021 for the UK and 2022 for Scotland. I don’t know why there’s this extra little lag. It might just be kind of a fluke honestly. But so that’s how we got employment. To get inactivity and unemployment, that’s like an extra leap. Basically, anyone who’s working aged, who is not in employment automatically is either unemployed or inactive. To get unemployment numbers, again, there’s really nothing. The assumption that we made was that the ratio between unemployment and inactive people in the LFS was, we just copied that over.

Again, that’s a pretty big assumption. It’s probably not super off base.

You can look at some administrative data on uh, the number of people that take, like, Jobseekers’ Allowance or work related claimant count benefits and the ONS does publish that. And again that is administrative data.

Not everyone who’s unemployed will take those benefits. Not everyone taking those benefits will be classed as unemployed, kind of some needle-y definitions there. But it looks, it, it mirrors the official unemployment figures pretty well for a couple years and then they diverge again. So it’s sort of a hard thing to use but – And in the end, when we did this modelling, we didn’t use that for unemployment. We just used this ratio. But that’s one thing that you could kind of get at.

For inactivity there’s, there’s just nothing. There’s no way to understand inactivity without the Labour Force Data survey and making some of these broad assumptions about the economy.

00:16:24 Hannah Randolph

So the use of this administrative data to understand what’s going on, I mean, it is telling you something about the labour market more generally, even if it’s not perfectly representing of these statistics that we’re used to having.

But it does sound like in terms of those you know, particularly unemployment statistics, this is really a stopgap measure until the LFS quality improves. Is that fair to say?

00:16:46 Allison Catalano

Yeah, it is, and it looks like, you know, we’ve kept up this modelling through the most recent data, which I think is gonna be  May-ish, May or June. The most recent data would have just come out mid August and it seems like this gap is closing. So since this independent review we talked about earlier came out, the ONS has had a little bit of a reshuffle and has sort of really changed some of their funding priorities and has increased their survey responses a little bit.

So the LFS and the model data that we have kind of come back together a little bit. They’re still pretty significant gaps, but it’s a long term thing. So we’ll kind of see where that goes.

00:17:35 Hannah Randolph

Yeah, because it’s – the way the LFS responses work is they survey the same people over five quarters, so I did have a look at how many interviews they’ve done. And so in the third quarter of 2023, which is around the time that the statistics lost their accreditation, they completed 35,000 person interviews in the UK, that includes Northern Ireland, but that’s now gone up to 59,000 in the first quarter of 2025. So they have been able to, they have achieved more person interviews and so, because they have to interview people over a number of quarters, it does take a while for that to kind of feed through to the full survey and all of the survey numbers and then the APS and stuff like that.

00:18:23 Allison Catalano

In the meantime, one of the issues is honestly, like misinformation around it. Because what we’re seeing with employment levels, most, but again, it’s the thing that we can model the best, what it looked like is that there was this big U, this big dip, where suddenly out of nowhere employment was dropping and now suddenly out of nowhere, employment is coming up again.

And you know, policymakers and people I’ve been in conversations with are like, ohh, employment’s going back up, that’s so great. What happened? And it seems like, well, nothing. The economy is kind of going in its direction. It actually seems to be coming down a little bit. But again, we’re getting further away from those figures on self-employment. So if the model is not not, you know, it’s, it’s good, but we don’t really know. So it looks like in the model, employment’s kind of coming down and levelling out, whereas in the LFS itself, we’re kind of coming out of this big trough of bad data. So it’s not that, ohh, employment is magically rising out of nowhere and it it had this crazy shock and dropped out of nowhere. It’s that it stopped counting things well, and now it’s starting to count things well again.

00:19:31 Hannah Randolph

And the trend in between is potentially not, not actually happening.

00:19:32 Allison Catalano

The trend in between is, who knows? Yeah, is is probably totally incorrect. That probably just didn’t exist.

00:19:39 Hannah Randolph

Yeah. And one of the things about this is just that in general, for a number of reasons, changes to surveys usually take a long time to implement, and part of that is about, you know, consistency and maintaining data series and comparability and that kind of thing and making sure the processes are right. And so the ONS, I think even prior to this had been working on a transformed LFS, a TLFS. And then when this started happening, they were kind of accelerating the transition to the TLFS and then they slowed down a bit. What do you know about that process and and what they’ve been doing?

00:20:12 Allison Catalano

So basically, the long story is that the way that the LFS is administered is, it’s someone calls you and then shows up at your door and knocks on your door. Some combination of those two things. But basically someone is in your face telling you, asking you very politely to fill out a survey. And people increasingly don’t really want to do that.

So it had been slowing down for a while, where they’ve been getting less and less, fewer and fewer responses. In Northern Ireland, they’ve had success bringing in elements of online surveys. I think it’s, so they’ve had some elements, some success bringing in elements of online surveys and have, like I said, kept robust response rates this whole time. So in the UK they kind of – in Great Britain, they kind of started, they decided what they would do is they would roll out a fully online survey and they would stop with some of this, like knock-to-nudge where they’re showing up at your door, knocking, asking you to fill out the survey. So that was going to be the transformed Labour Force Survey. I don’t, I, I don’t remember when it started. I think sometime between 2023 and 2024 it was supposed to be rolled out.

And it was like the LFS will be no more. The TLFS is the new thing and then increasingly it got delayed and delayed and delayed and delayed. And now the last thing I heard was just from a BBC article a couple of months ago that’s, that – basically it’s, it’s gone, you know, by the wayside and that they’re not going to be doing that anymore, the move from face to face, from face to face to online questionnaires just didn’t go well or correctly.

So they’re going to start bringing in some online questionnaire elements but maintaining the face to face interviews because it seems like that’s the only way to get people to actually respond to the survey.

00:22:06 Hannah Randolph

I think some of it was about, you know, increasing incentives for people to respond and also trying to shorten the survey where they can, making it a little bit more modular, because that was part of it was, you know, the original survey when it started was had more limited questions. And then things got added on and then you know it takes hours to respond and, and people don’t want to do it.

00:22:27 Allison Catalano

Yeah. And, you know, one of the problems with that too is – so there’s actually, I think, in Great Britain, there’s really two survey areas. There’s Great Britain and then there’s Scotland north of the Caledonian Canal, which, you know, is pretty rural, sparsely populated, the Highlands. So there’s not very many people living up there. And to get a good, you know, volume of data about that population you’re having to interview them a lot. So one of the concerns that’s come up is just this, this idea of survey fatigue, where people are surveyed so much for so many things. It comes up in Northern Ireland too, a little bit, Northern Ireland can be quite, you know, protective about how much businesses are surveyed.

There’s, you know, it’s less than 2 million people live in Northern Ireland and the number of, they do a really good job surveying businesses, but there’s only so many times you can survey a business before they’re like, please leave me alone. And it’s been the same kind of north of the Caledonian Canal where, you know, the volume of surveying to get an idea of what’s going on in Western Isles and Shetland is really important, but it is, it can be quite a lot.

00:23:33 Hannah Randolph

So before we wrap up, Allison, is there anything else that you would like to talk about with regards to the LFS?

00:23:40 Allison Catalano

Yeah. I mean, I think, I think a lot of what, you know, we’ve said about the LFS, the the real issue that I think we come onto is, what are we going to do with policy? Like, how are policymakers supposed to make decisions given this?

And right now, the worry is that – and people are continuing to just trust these statistics blindly and are making the wrong decision or putting resource in the wrong places. So really, you know, my concern because of the area that I work in is with people who work in employability services to deliver really important programmes, you know, across the UK, but especially in Scotland, the way that they make decisions is entirely based on this. They, they, the only way that they know who to target is going to be based on this sort of data source.

For people working in employability programmes, there’s always this worry of like, we feel like we’re not capturing the right people. How do we get the right people? And the only way that they can get the right people is if they know who they’re not capturing. And the only way that they know who they’re not capturing is through data that is, that doesn’t exist right now. So you know from a fiscal point, it’s really worrying that we, how do we direct resources appropriately if we don’t know if we’re flying blind, essentially. So yeah, right now that’s sort of my, my big worry. And I mean I’m looking forward to seeing the LFS improve a little bit in the future, I’m hoping that we’ll be able to get some of this sorted out, but also I’m, I’m mindful of, you know, some of the issues with administrative data as well. And I do wonder if there are things that we can learn from the LFS, you know, this really, fiasco, this like absolute catastrophe for economic statistics. Can we do things to make this better? Can we make pay data better? Can we make self-employment data a little bit more responsive? Is there anything that we can do to make it so that we have a backup in the future, a reliable backup in the future. So I don’t know.

If anyone from the ONS is listening, get – we’re sorry, but get on it, man.

00:25:57 Hannah Randolph

Yeah. And, you know, one of the challenges too, like you were saying with employability programmes in Scotland, those are delivered at a local level. So it does seem like there’s a good understanding within Scottish Government, particularly the statisticians and analysts of the challenges and issues with these data. But then if you get down to the, the sort of local level you’re asking a lot of people to have all of this like statistical expertise.

00:26:20 Allison Catalano

And know the gossip in the ONS! Which they, they wouldn’t. And, and even on a like – the analysts in Scottish Government are obviously very aware of what is going on and are working really hard to get to a point where like, they know what’s happening in the economy. But politicians don’t know! People working at a low level don’t know! People working in industries that are, you know, rely on this as well, don’t know necessarily what’s going on. And yeah, I think it’s worrying. Obviously it’s very –

00:26:49 Hannah Randolph

Yeah, there’s been this kind of push of like, OK, you should look at a suite of indicators and not just the headline, like employment, unemployment statistics. But again, that’s, that’s putting more of a burden on people of, kind of, you should know about a range of indicators and what they mean.

00:27:05 Allison Catalano

And yeah, and where to find them and just trust that those are reliable, even though these other ones aren’t reliable and it’s, it’s really, it’s a hard – it’s hard to make decisions in in a vacuum, in the vacuum that we’re in right now. Yeah. And it’ll be hard to know, you know, just overall how we’re performing in, in the future.

00:27:24 Hannah Randolph

So thank you for joining me today, Allison. We’ll be following this fairly closely and updating as changes to the LFS come in and we’ll see if the quality of the statistics improves. We hope that it improves. Until the next Fraser of Allander podcast, we’ll see you then.

00:27:38 Allison Catalano

Thanks, Hannah.

00:27:49 Ben Cooper

Hello, 2025 marks a significant year for the Fraser of Allander Institute. Founded in 1975, the Institute celebrates 50 years of leading economic research in Scotland. Given this, we have a series of exciting events and content planned for the remainder of this year to mark this milestone.

This includes our anniversary conference on the 18th and 19th of September this year at the University of Strathclyde’s Technology and Innovation Centre. We have a number of speakers and themed sessions planned for the day in order to highlight the breadth and importance of the Economic Research being conducted across Scotland.

As well as this we have a number of other events planned, including our reception at the Scottish Parliament in December and one at the Scotland Office in London in October.

You’ll also see some unique content published in the coming months, including a podcast series with past institute directors and some articles on the history of the Institute, as well as some other eye-catching covers from some of our reports from over the years in the coming months.

In order to be kept up to date with any of these events and to be the first to know about our 50th anniversary publications, visit our website, fraserofallander.org and you can join our mailing list.

If you are interested in engaging with us also, whether you previously work for or with the institute, why not get in touch? We have a designated 50th anniversary e-mail fraser50@strath.ac.uk. We hope you’re as excited as we are and look forward to celebrating this exciting milestone with as many of you as possible.