Why ATOS are getting it wrong: Part 4 - Fallacies and starting positions

4.Base rate fallacy, prosecutor's fallacy, and neglecting the claimant's history

When a claimant arrives for an ATOS assessment, the assessor pretty much seems to view them as a blank slate with no history. So a person who has been disabled or ill and on incapacity benefit or ESA for many years starts from a position of no points he same way as someone who has only just claimed.

The first reason this is problematic is due to the way the criteria for ESA are now so stringent, that even someone who is clearly very disabled indeed only just scrapes through. This is especially likely to be the case if their functional problems are limited to just one area. The maximum that person can score is 15 points and the number of points they need is also 15. In other words there is no margin for error at all. But as with anything in the real world, there will inevitably be an error rate. As I discussed in an earlier post, if your error rate for a given assessment is 10% (which seems generously low) then this approaches 100% if you reassess the same claimant enough times.

This leads to the second problem. By its very nature, it is likely that most people on ESA will be entitled to it. This is because they need to, and have probably been advised to, claim it. They've also got through the initial claim stages, and have been signed off work by a doctor. They're therefore a self-selected sample. Although most people in the population are not entitled to ESA, the vast majority of them are also not claiming it. The confusion arises when assessors see claimants as no more likely to meet the criteria for ESA than any other member of the population, so start from a position that the person being assessed will likely fail. This is a form of the base rate fallacy, and biases the assessment. It's also a form of the prosecutor's fallacy.

Consider another example: you know that only 5% of teapots are valuable. 95% are worthless. You train as an assessor of teapots, and are asked to assess the teapots in someone's collection. Let's assume in this scenario that assessing teapots on appearance is possible, but notoriously difficult and you know you will often get it wrong.

What are the most important things you need to know about each of the teapots you assess?

• Its history - The process by which it got into that collection.
• What previous assessments were made of it that got it there?
• What evidence was used in those assessments?
• How reliable is that evidence?

The answers to these questions radically alters the likelihood that you should trust your own judgment that a given teapot appears at first glance to be valuable or not. And the odds that a given teapot in someone's collection will be worthless are most definitely not 95%.

Do ATOS assessors understand these issues? What do you think?

The solution

It's really quite easy. When a claimant is being re-assessed, they start with the number of points they were given at the previous assessment (including any appeal). Points can be added or removed, but the change must be specifically identified, and evidenced, in enough detail. Likewise, if someone was previously put in the support group, they start in it in the same way. This should have the effect of focusing the assessor's mind better onto what should be reasonably expected in that case.

This might not solve all the other problems but it should massively reduce the error rate caused by constant re-assessment. It should also satisfy IDS and Grayling's desire not to 'write people off', misguided as that is.