Why ATOS are getting it wrong: Part 4 - Fallacies and starting positions

4.Base rate fallacy, prosecutor's fallacy, and neglecting the claimant's history

When a claimant arrives for an ATOS assessment, the assessor pretty much seems to view them as a blank slate with no history. So a person who has been disabled or ill and on incapacity benefit or ESA for many years starts from a position of no points he same way as someone who has only just claimed.

The first reason this is problematic is due to the way the criteria for ESA are now so stringent, that even someone who is clearly very disabled indeed only just scrapes through. This is especially likely to be the case if their functional problems are limited to just one area. The maximum that person can score is 15 points and the number of points they need is also 15. In other words there is no margin for error at all. But as with anything in the real world, there will inevitably be an error rate. As I discussed in an earlier post, if your error rate for a given assessment is 10% (which seems generously low) then this approaches 100% if you reassess the same claimant enough times.

This leads to the second problem. By its very nature, it is likely that most people on ESA will be entitled to it. This is because they need to, and have probably been advised to, claim it. They've also got through the initial claim stages, and have been signed off work by a doctor. They're therefore a self-selected sample. Although most people in the population are not entitled to ESA, the vast majority of them are also not claiming it. The confusion arises when assessors see claimants as no more likely to meet the criteria for ESA than any other member of the population, so start from a position that the person being assessed will likely fail. This is a form of the base rate fallacy, and biases the assessment. It's also a form of the prosecutor's fallacy.

Consider another example: you know that only 5% of teapots are valuable. 95% are worthless. You train as an assessor of teapots, and are asked to assess the teapots in someone's collection. Let's assume in this scenario that assessing teapots on appearance is possible, but notoriously difficult and you know you will often get it wrong.

What are the most important things you need to know about each of the teapots you assess?

• Its history - The process by which it got into that collection.
• What previous assessments were made of it that got it there?
• What evidence was used in those assessments?
• How reliable is that evidence?

The answers to these questions radically alters the likelihood that you should trust your own judgment that a given teapot appears at first glance to be valuable or not. And the odds that a given teapot in someone's collection will be worthless are most definitely not 95%.

Do ATOS assessors understand these issues? What do you think?

The solution

It's really quite easy. When a claimant is being re-assessed, they start with the number of points they were given at the previous assessment (including any appeal). Points can be added or removed, but the change must be specifically identified, and evidenced, in enough detail. Likewise, if someone was previously put in the support group, they start in it in the same way. This should have the effect of focusing the assessor's mind better onto what should be reasonably expected in that case.

This might not solve all the other problems but it should massively reduce the error rate caused by constant re-assessment. It should also satisfy IDS and Grayling's desire not to 'write people off', misguided as that is.

Why ATOS are getting it wrong - Part 3

3. Misinterpretation of descriptors

This is something Greg Wood has talked about a lot, so I won't repeat it all in so much detail. I highly recommend his posts on the subject - he is completely correct to assert that this is an area of significant concern. The descriptors for ESA are here: to get or retain ESA, the claimant must score 15 points. Points are combined, but only one descriptor from a single activity can apply at once.

Although the LIMA system seems to award or rule out some descriptors, it is also clear from the manual that the assessor has a certain level of input as to when to award a descriptor as well. Since whether a descriptor applies is a legal question and not a matter of personal discretion, it is vital that the correct legal test is applied.

One particular problem seems to be what I would call an improper 'all or nothing' test. This is where if the assessor rules out the highest descriptor but doesn't appear to bother to check whether a lower one might apply. For example, a claimant states they can manage to see family every day. No points are awarded for activity 16 - dealing with other people. But hang on: what about 16 (b) and 16 (c) - these could still apply, as they relate to unfamiliar people, so more information is needed before ruling them out.

Similarly, I've seen monthly blackouts scoring no points, despite apparently scoring 6 points for activity 10. The assessor claimed that monthly was not significant enough to score points - but that isn't right - it's not what the descriptors say. A personal view of what's 'significant' can't override the clear wording of the law, but that seems to be what is happening.


The converse of 'all or nothing' is the 'bottom up counting' error. This occurs when the findings suggest that a high level descriptor applies - maybe one scoring 15 points - but instead a lower descriptor is awarded that is also applicable. For example, if a claimant has two blackouts a week, both descriptors 10 (a) and 10 (b) apply. Since only one of them can be awarded, the higher of the two should be awarded. 'Bottom up counting' leads to the lower wrongly being awarded.


A person who should have scored 15 points from a single descriptor thus ends up with fewer, and is kicked off ESA. I've seen this happen to several people.

Why ATOS are getting it wrong - Part 2

2. LIMA, and abuse of the concept of the typical day



When ATOS carry out an assessment, the assessor uses a computer system called LIMA. The manual is available here: although it's an old version from the incapacity benefit days, it explains a great deal to anyone who has read an ESA report, so it obviously works in a similar way still.



The key thing to note is the way the 'typical day' box is used. The assessor asks the claimant what he or she does on a typical day, and the computer extracts keywords from this to suggest which descriptors might apply. The computer rules out descriptors that cannot apply due to the claimant's abilities on a typical day.



At this point, it is worth noting how the law views ability and inability. Since a case called Moyna, it is pretty well established that a broad view must be taken of fluctuating conditions, so that a person who can manage things only occasionally is not deemed able to do them often enough to count as able to do them at all. If a person is usually able to manage things, on the other hand, he or she is deemed able to do them.



Therefore the ATOS concept of the typical day is a potentially valid one, provided that activities entered here really do represent activities managed on a typical day. As you may have guessed, this is the problem.



I've seen an 'ability' to see a GP every 6 weeks entered in the 'typical day' box. This was then extrapolated by LIMA to deny the claimant points for ability to deal with unfamiliar people. There was also no finding that he managed even those 6-weekly occasions in a reasonable manner without getting too upset, contrary to further case law on what is meant by 'ability'.



Another common thing entered in the 'typical day' box is: turning up to the ATOS assessment! This is used to deny a range of descriptors, with incorrect findings (or no findings) as to the repeatability of having to do this on a daily basis, or how well the claimant coped with it on the one occasion, or whether excessive pain resulted. Other offending entries here are occasional hobbies, visits to family, cooking a proper meal once a week or fortnight, and going shopping once a week or fortnight.



No wonder that all of these things being done at once on a 'typical day' results in an adverse finding. Yet an ability to do those things occasionally is perfectly consistent with quite severe limitations.

Why ATOS assessments get it so badly wrong - Part 1

I've been back working on ESA appeals, representing appellants, for a few months again now.

In my previous post I looked at how ministers are very confused about what role ESA is supposed to have, and how this is causing some of the problems with it.

Now, in a series of posts, I'm going to look at the far more familiar problem of ATOS assessments being of very questionable quality in and of themselves, and will attempt to explain why I think this is the case.

Contrary to the popular view, I don't think that the explanation is that the government pays a bonus to find against claimants. Although this wouldn't necessarily surprise me, there are enough other problems that this additional factor isn't needed to explain what's going on. The present setup already makes sense without it.

Here's part one of why there are serious problems:-

1. Constant reassessment means errors are inevitable

When I was on ESA, I questioned the DWP's intention to reassess me after one year, as the condition I had was not ever likely to change and would always present the same barriers to work. The answer I received was that in effect I should be grateful, as most people are reassessed after three or six months.

In the previous post linked to above, I discussed why the government's explanation of why it thinks constant reassessment is good is flawed. ESA is supposed to support people with limitations towards work, so kicking them off it is clearly not conducive to providing that support.

The problem with constant reassessment is that it makes errors inevitable in the long term. If the error rate so that an ill claimant's ESA is wrongly stopped is 10%, then over the course of a  lifetime, a chronically ill person faces a probability approaching 100% of having their ESA stopped wrongly at some point. Having this happen even once is catastrophic for that person.

In practice the error rate is almost certainly higher than 10%.

ESA and 'fit for work'

It is very common to see a statement that some whose ESA has stopped has been found 'fit for work'. This is not really correct and is worth examining in some detail.

I think the problem is that the people at the top, who should know what they're doing with it,  have become very confused about what role ESA is supposed to have.

When it was introduced, the stated aim was that it was supposed to be a system to support those with barriers into work, rather than 'writing them off' -  as, it was claimed, the old system did. This was the whole point of having the work-related activity group, into which most people on ESA would go. People in it would be encouraged to eventually find work but with their barriers recognised. In other words, people on ESA would *not* be seen as 'written off'.

Section 8 of the Welfare Reform Act 2007 bears this out: the test for eligibility is not whether a person is incapable of any work, but whether their capability for work is limited, such that it is not reasonable to *require* them to work. It is easy to imagine a situation where a person could work but due to disability it is not reasonable to require them to: it might causes pain or psychological distress, or could risk a relapse. The test in the primary legislation is therefore, on paper, less strict than being a need to be 'incapable' of work.

But when the first ESA criteria were published in 2008, it was obvious to everybody that the level of disability needed to get even into the WRAG was more severe than under IB. This has led to the bizarre and cruel situations where people are having their ESA stopped after medical assessments even when it's obvious that it would not be reasonable to require them to work. (Problems with wild speculation about people's abilities made at ATOS assessments do compound this further, but the criteria that ATOS have to apply must share the blame.)

Purnell, and now IDS and Grayling, have justified this by claiming that it is actually better for people to have their ESA stopped, and that being on ESA is to be 'written off'. This is where the confusion really starts. If you recall, this ignores, and in fact reverses, the very purpose that ESA was stated to have: that people on it were not supposed to be being 'written off' as the benefit was supposed to be more like a lighter form of JSA, and not just a clone of IB. But they've always talked about it as though it were just a clone of IB: hence the misleading headlines that people have been found 'fit fork work' when their ESA is stopped. Even welfare campaigners have bowed to the inevitable and have started to adopt the 'fit for work' terminology as a shorthand, driving the vicious cycle of misunderstanding.

The situation has got even more confused with some of the latest amendments, where the actual stated reasoning behind them has been to exclude people from ESA if they could work at all, even in some highly theoretical scenarios. Hence now being unable to walk but able to push a wheelchair 50m, but not 100m, precludes entitlement to ESA at all, unless combined with another problem. Of course there are jobs that could be done, but how (for example) is the person supposed to get to work in the first place and to get home again, every day, and on time?

Similarly, a person who can get only to familiar places and only with a support worker scores only 9 points and is precluded from ESA, unless there is another problem in combination with it to get them up to 15. But the number of available mental health descriptors has been cut down to avoid so-called 'double counting'. 9 points gets the person nothing and they are 'fit for work' with full JSA conditionality. Although again, there would be jobs the person could do, how would they avoid falling foul of the (it seems) universally accepted system of job selection by interview, when turning up frightened and with a support worker? Yet ESA potentially does not even recognise this as a limitation at all any more.

I would very much like to see the 'fit for work' terminology challenged as well, but due to the very great confusion at the top over what ESA is, it is somewhat realistic under the current criteria of very severe incapacity. But to acknowledge this is also to acknowledge that ESA has failed its most basic stated aim of reform, as almost by definition anyone able to receive it at all has been 'written off'.