Why ATOS are getting it wrong: Part 4 - Fallacies and starting positions

4.Base rate fallacy, prosecutor's fallacy, and neglecting the claimant's history

When a claimant arrives for an ATOS assessment, the assessor pretty much seems to view them as a blank slate with no history. So a person who has been disabled or ill and on incapacity benefit or ESA for many years starts from a position of no points he same way as someone who has only just claimed.

The first reason this is problematic is due to the way the criteria for ESA are now so stringent, that even someone who is clearly very disabled indeed only just scrapes through. This is especially likely to be the case if their functional problems are limited to just one area. The maximum that person can score is 15 points and the number of points they need is also 15. In other words there is no margin for error at all. But as with anything in the real world, there will inevitably be an error rate. As I discussed in an earlier post, if your error rate for a given assessment is 10% (which seems generously low) then this approaches 100% if you reassess the same claimant enough times.

This leads to the second problem. By its very nature, it is likely that most people on ESA will be entitled to it. This is because they need to, and have probably been advised to, claim it. They've also got through the initial claim stages, and have been signed off work by a doctor. They're therefore a self-selected sample. Although most people in the population are not entitled to ESA, the vast majority of them are also not claiming it. The confusion arises when assessors see claimants as no more likely to meet the criteria for ESA than any other member of the population, so start from a position that the person being assessed will likely fail. This is a form of the base rate fallacy, and biases the assessment. It's also a form of the prosecutor's fallacy.

Consider another example: you know that only 5% of teapots are valuable. 95% are worthless. You train as an assessor of teapots, and are asked to assess the teapots in someone's collection. Let's assume in this scenario that assessing teapots on appearance is possible, but notoriously difficult and you know you will often get it wrong.

What are the most important things you need to know about each of the teapots you assess?

• Its history - The process by which it got into that collection.
• What previous assessments were made of it that got it there?
• What evidence was used in those assessments?
• How reliable is that evidence?

The answers to these questions radically alters the likelihood that you should trust your own judgment that a given teapot appears at first glance to be valuable or not. And the odds that a given teapot in someone's collection will be worthless are most definitely not 95%.

Do ATOS assessors understand these issues? What do you think?

The solution

It's really quite easy. When a claimant is being re-assessed, they start with the number of points they were given at the previous assessment (including any appeal). Points can be added or removed, but the change must be specifically identified, and evidenced, in enough detail. Likewise, if someone was previously put in the support group, they start in it in the same way. This should have the effect of focusing the assessor's mind better onto what should be reasonably expected in that case.

This might not solve all the other problems but it should massively reduce the error rate caused by constant re-assessment. It should also satisfy IDS and Grayling's desire not to 'write people off', misguided as that is.

Why ATOS are getting it wrong - Part 3

3. Misinterpretation of descriptors

This is something Greg Wood has talked about a lot, so I won't repeat it all in so much detail. I highly recommend his posts on the subject - he is completely correct to assert that this is an area of significant concern. The descriptors for ESA are here: to get or retain ESA, the claimant must score 15 points. Points are combined, but only one descriptor from a single activity can apply at once.

Although the LIMA system seems to award or rule out some descriptors, it is also clear from the manual that the assessor has a certain level of input as to when to award a descriptor as well. Since whether a descriptor applies is a legal question and not a matter of personal discretion, it is vital that the correct legal test is applied.

One particular problem seems to be what I would call an improper 'all or nothing' test. This is where if the assessor rules out the highest descriptor but doesn't appear to bother to check whether a lower one might apply. For example, a claimant states they can manage to see family every day. No points are awarded for activity 16 - dealing with other people. But hang on: what about 16 (b) and 16 (c) - these could still apply, as they relate to unfamiliar people, so more information is needed before ruling them out.

Similarly, I've seen monthly blackouts scoring no points, despite apparently scoring 6 points for activity 10. The assessor claimed that monthly was not significant enough to score points - but that isn't right - it's not what the descriptors say. A personal view of what's 'significant' can't override the clear wording of the law, but that seems to be what is happening.


The converse of 'all or nothing' is the 'bottom up counting' error. This occurs when the findings suggest that a high level descriptor applies - maybe one scoring 15 points - but instead a lower descriptor is awarded that is also applicable. For example, if a claimant has two blackouts a week, both descriptors 10 (a) and 10 (b) apply. Since only one of them can be awarded, the higher of the two should be awarded. 'Bottom up counting' leads to the lower wrongly being awarded.


A person who should have scored 15 points from a single descriptor thus ends up with fewer, and is kicked off ESA. I've seen this happen to several people.

Why ATOS are getting it wrong - Part 2

2. LIMA, and abuse of the concept of the typical day



When ATOS carry out an assessment, the assessor uses a computer system called LIMA. The manual is available here: although it's an old version from the incapacity benefit days, it explains a great deal to anyone who has read an ESA report, so it obviously works in a similar way still.



The key thing to note is the way the 'typical day' box is used. The assessor asks the claimant what he or she does on a typical day, and the computer extracts keywords from this to suggest which descriptors might apply. The computer rules out descriptors that cannot apply due to the claimant's abilities on a typical day.



At this point, it is worth noting how the law views ability and inability. Since a case called Moyna, it is pretty well established that a broad view must be taken of fluctuating conditions, so that a person who can manage things only occasionally is not deemed able to do them often enough to count as able to do them at all. If a person is usually able to manage things, on the other hand, he or she is deemed able to do them.



Therefore the ATOS concept of the typical day is a potentially valid one, provided that activities entered here really do represent activities managed on a typical day. As you may have guessed, this is the problem.



I've seen an 'ability' to see a GP every 6 weeks entered in the 'typical day' box. This was then extrapolated by LIMA to deny the claimant points for ability to deal with unfamiliar people. There was also no finding that he managed even those 6-weekly occasions in a reasonable manner without getting too upset, contrary to further case law on what is meant by 'ability'.



Another common thing entered in the 'typical day' box is: turning up to the ATOS assessment! This is used to deny a range of descriptors, with incorrect findings (or no findings) as to the repeatability of having to do this on a daily basis, or how well the claimant coped with it on the one occasion, or whether excessive pain resulted. Other offending entries here are occasional hobbies, visits to family, cooking a proper meal once a week or fortnight, and going shopping once a week or fortnight.



No wonder that all of these things being done at once on a 'typical day' results in an adverse finding. Yet an ability to do those things occasionally is perfectly consistent with quite severe limitations.

Why ATOS assessments get it so badly wrong - Part 1

I've been back working on ESA appeals, representing appellants, for a few months again now.

In my previous post I looked at how ministers are very confused about what role ESA is supposed to have, and how this is causing some of the problems with it.

Now, in a series of posts, I'm going to look at the far more familiar problem of ATOS assessments being of very questionable quality in and of themselves, and will attempt to explain why I think this is the case.

Contrary to the popular view, I don't think that the explanation is that the government pays a bonus to find against claimants. Although this wouldn't necessarily surprise me, there are enough other problems that this additional factor isn't needed to explain what's going on. The present setup already makes sense without it.

Here's part one of why there are serious problems:-

1. Constant reassessment means errors are inevitable

When I was on ESA, I questioned the DWP's intention to reassess me after one year, as the condition I had was not ever likely to change and would always present the same barriers to work. The answer I received was that in effect I should be grateful, as most people are reassessed after three or six months.

In the previous post linked to above, I discussed why the government's explanation of why it thinks constant reassessment is good is flawed. ESA is supposed to support people with limitations towards work, so kicking them off it is clearly not conducive to providing that support.

The problem with constant reassessment is that it makes errors inevitable in the long term. If the error rate so that an ill claimant's ESA is wrongly stopped is 10%, then over the course of a  lifetime, a chronically ill person faces a probability approaching 100% of having their ESA stopped wrongly at some point. Having this happen even once is catastrophic for that person.

In practice the error rate is almost certainly higher than 10%.

ESA and 'fit for work'

It is very common to see a statement that some whose ESA has stopped has been found 'fit for work'. This is not really correct and is worth examining in some detail.

I think the problem is that the people at the top, who should know what they're doing with it,  have become very confused about what role ESA is supposed to have.

When it was introduced, the stated aim was that it was supposed to be a system to support those with barriers into work, rather than 'writing them off' -  as, it was claimed, the old system did. This was the whole point of having the work-related activity group, into which most people on ESA would go. People in it would be encouraged to eventually find work but with their barriers recognised. In other words, people on ESA would *not* be seen as 'written off'.

Section 8 of the Welfare Reform Act 2007 bears this out: the test for eligibility is not whether a person is incapable of any work, but whether their capability for work is limited, such that it is not reasonable to *require* them to work. It is easy to imagine a situation where a person could work but due to disability it is not reasonable to require them to: it might causes pain or psychological distress, or could risk a relapse. The test in the primary legislation is therefore, on paper, less strict than being a need to be 'incapable' of work.

But when the first ESA criteria were published in 2008, it was obvious to everybody that the level of disability needed to get even into the WRAG was more severe than under IB. This has led to the bizarre and cruel situations where people are having their ESA stopped after medical assessments even when it's obvious that it would not be reasonable to require them to work. (Problems with wild speculation about people's abilities made at ATOS assessments do compound this further, but the criteria that ATOS have to apply must share the blame.)

Purnell, and now IDS and Grayling, have justified this by claiming that it is actually better for people to have their ESA stopped, and that being on ESA is to be 'written off'. This is where the confusion really starts. If you recall, this ignores, and in fact reverses, the very purpose that ESA was stated to have: that people on it were not supposed to be being 'written off' as the benefit was supposed to be more like a lighter form of JSA, and not just a clone of IB. But they've always talked about it as though it were just a clone of IB: hence the misleading headlines that people have been found 'fit fork work' when their ESA is stopped. Even welfare campaigners have bowed to the inevitable and have started to adopt the 'fit for work' terminology as a shorthand, driving the vicious cycle of misunderstanding.

The situation has got even more confused with some of the latest amendments, where the actual stated reasoning behind them has been to exclude people from ESA if they could work at all, even in some highly theoretical scenarios. Hence now being unable to walk but able to push a wheelchair 50m, but not 100m, precludes entitlement to ESA at all, unless combined with another problem. Of course there are jobs that could be done, but how (for example) is the person supposed to get to work in the first place and to get home again, every day, and on time?

Similarly, a person who can get only to familiar places and only with a support worker scores only 9 points and is precluded from ESA, unless there is another problem in combination with it to get them up to 15. But the number of available mental health descriptors has been cut down to avoid so-called 'double counting'. 9 points gets the person nothing and they are 'fit for work' with full JSA conditionality. Although again, there would be jobs the person could do, how would they avoid falling foul of the (it seems) universally accepted system of job selection by interview, when turning up frightened and with a support worker? Yet ESA potentially does not even recognise this as a limitation at all any more.

I would very much like to see the 'fit for work' terminology challenged as well, but due to the very great confusion at the top over what ESA is, it is somewhat realistic under the current criteria of very severe incapacity. But to acknowledge this is also to acknowledge that ESA has failed its most basic stated aim of reform, as almost by definition anyone able to receive it at all has been 'written off'.

Examining two of Lord Freud's assurances on ESA in 2011

1. In a Lords debate in May 2011, Lord Freud said this:

The regulations do not specify the type of activity that we expect this group to engage in. Indeed, the definition of work-related activity is deliberately broad, covering any activity which makes it more likely that the person will obtain or remain in work. However, there are some absolutes. We will not require customers to undertake medical treatment nor to seek, apply for or take up work. Beyond that, we do not want to be overly prescriptive.

By November 2011, he was saying this, in a debate on the very next Welfare Reform Bill:

Noble Lords asked, in relation to Clause 16, whether this measure extends the definition of work-related activity, which is one of the questions asked by the noble Lord, Lord McKenzie. The Bill seeks to clarify what may be included by way of work-related activity, rather than extend its meaning. Work-related activity is already defined in the Welfare Reform Act 2007 as,

"activity which makes it more likely that the person will obtain or remain in work or be able to do so",

and Clause 54 makes expressly clear that this may include work experience or a work placement.

So in under six months, an 'absolute' limit on never requiring ESA claimants to take up work had already been  abandoned. This is just about the very soonest that change to the law could possibly have been implemented, in express violation of Freud's assurance.

Moving on....

2.  In the exact same May 2011 debate, Freud gave this assurance:-

Customers who do not participate in work-related activity will be sanctioned, and I know noble Lords have some concerns in this area. In response, let me just say this: there are safeguards in place to ensure that sanctions are not improperly applied. Sanctions will apply only to the work-related activity component of the benefit

This time it took until December 2012, but again the exact opposite has been put into place:

What is changing?

Under the new rules ESA claimants in the WRAG who fail to comply with the conditions for receiving benefit will receive an open ended sanction, followed by a fixed period sanction when they re-comply. The fixed period sanction will be one week for a first failure, two weeks for a second failure and four weeks for a third and subsequent failures in a 52 week period.

Claimants who are sanctioned will lose all of their personal allowance, but their work related activity component will not be affected.

In other words, the sanction has been moved from the smaller WRA component of about £30 to the personal allowance of £71, so has increased 230% from the level of Freud's promised safeguard limit.

What this shows is that nothing the government says about plans for benefits can be relied on. Any assurance or promise of safeguards is completely worthless and unenforceable. Although the second of these examples did not happen for over a year, the amendment was planned, drafted, and notified to claimants a lot earlier, so in practice the change started almost the moment Freud promised the opposite.

Proposed changes to child benefit - a reasonable idea turned nightmarish

Iain Duncan Smith has proposed limiting benefits payable for children in any family to a maximum of two children. If the parent or parents then went on to have more children, they would not receive additional money for the third or higher child.

In some ways, this is quite a reasonable idea. The additional amounts of benefit paid for families with more children can be fairly generous. Each additional child can currently bring in approximately £70pw in tax credits and child benefit. It also entitles the family to rent a larger house and to a higher income disregard in housing and council tax benefit worth about £50pw for each child.

I have thought for some time that some kind of limit to how many children a person can have while claiming benefits, just as someone who wants to claim help for a mortgage cannot usually take out a new mortgage while on benefit.

The extent of additional money for each child has already been somewhat limited by the new, and highly flawed, benefit cap of £500 a week per household. That system will only affect people with 4-6 children, depending on a number of factors such as the amount of rent they pay. If this new change were implemented, then it would make the cap completely obsolete as nobody with under four children would ever be affected by it. But this is where we get into difficulties with what has now been proposed.

The first problem with the new proposal is that Duncan Smith is once again making it into an issue of those in work against those out of work. From his description of things, you would get the impression that 'those with three or more children' and 'those in work' are two separate groups on a Venn diagram that don't overlap. But this is wrong - people in work can obviously have more than two children.

Despite purporting to make this distinction, there is nothing in IDS's announcement that suggests this change will be limited to those out of work. It seems that everyone will be affected. So it's not just something aimed at the unemployed - we need to be clear on that, or a falsehood is spread.

As I said above, I think it would be reasonable to limit people from having more children while on benefits and immediately claiming more money for them. But it would definitely not be fair to say that if a parent with 4 children loses his or her job, then he or she is unable to claim enough money to live on. It's not as though someone who has lost a job can give up children.

And this is the point where fridge horror sets in. Benefits for children include help with higher rent, for the larger family. This proposal means that a parent with 4 children losing his job would almost immediately become both homeless and un-houseable. This completely undermines the idea of safety net. We may see families forced to split up and children being sent to live with other relatives (who will then be able to claim benefits for them - ruining the point of the idea once again.)

This government has no idea what it is doing. It just pushes through seriously flawed ideas without thinking them through, before and again. No-one who looked carefully at the last benefit cap thought it was a good idea, so I am very worried about this new idea.

Is the benefit cap really pushing thousands back into work?

Short answer: no. But a Daily Mail article nevertheless claims this is the case.

IDS claims that the upcoming benefit cap of £500 pw is causing 'thousands' of families to look for work. So that's already a bit of a climb down from them already going back into work, for a start. 1,700 people in this group have found work, which is 3% of the group. And we don't know over what time period that is: in any case it is a pretty low take-up rate.

The cap of £500 which starts in April 2013 will affect very few people. It will only apply to families with a lot of children. Families with a lot of children can indeed receive relatively high rates of benefits: after all, they have a lot of people to split that money between. On the other hand, it is not necessarily unreasonable to limit the number of children people can have and immediately request state support for them all.

But this is where the reasoning for this particular system of cap breaks down. It is wrong to compare it to the amount 'an average couple would earn in work'. This is for two reasons:

1. A family of the same size with the parents in work can also claim a lot in benefits - possibly the same or even more than one out of work. (This is even lampshaded in the Mail article!)
2. The £500 a week in benefits has to support between 5-8 people, whereas an average couple in work earning that only have to support two people. (And if they have children they will get additional money through benefits.)

The cap also creates a new 'couple penalty' - remember, that thing Cameron promised to abolish but hasn't? The cap applies per household. The obvious thing for a large family with two parents affected by the cap to do would be to split into two nearby households - who then only have a total cap of £1000 pw between them, which would be absolutely impossible ever to reach.

If the cap of a mere £500 per week really is causing 'thousands' people to look for work, how could IDS possibly know about it?

The only way would be where they are claiming jobseeker's allowance, the DWP would have a record of their jobsearch activity and requirements. But in that case, those jobsearch requirements would have already applied without the cap. Most families with children need to claim JSA, and to look for work as a basic condition of entitlement. That has been the law since the 1920s.

Look again at the exact words used:

 "Despite all the scaremongering, research now shows that of those housing benefit claimants affected by the cap, a third said they would now be looking for a job."

Note that it doesn't say this is new, or because of the cap. They could have been looking for a job all along - and must have been.