Autism and Evidence 6. What is wrong with the draft NICE guidance on “management” for children with autism?

What is wrong with the draft NICE guidance on “management” for children with autism?

I promised to post something on this blog about my thoughts on the NICE clinical guideline for autism in children currently out for consultation (http://www.nice.org.uk/guidance/index.jsp?action=folder&o=63399). This blog isn’t in the form of the response that I might submit to NICE partly because they ask for a response using a standard form rather than a text story. I’m also not going to write about details here – more ideas and broader issues.

As a starting point, I’d like to acknowledge how tough it must have been for the experts and other members of the committee who worked on this NICE guideline. It is pretty clear in the document that NICE procedures were rigorously (and rigidly) applied. This includes the use of the GRADE system that surely was not designed for the purpose of understanding the evidence for complex interventions in contexts that will not be the same as short primary or secondary healthcare consultation. For example, the rating of quality of evidence from any intervention included in this NICE autism review has been downgraded if the meta-analysis sample is under 400 children (because any smaller sample may not provide a good estimate of the size of effect of an intervention). This is despite some well-designed individual trials that passed strict peer review for competitive funding and were published in some of the highest impact scientific journals. Such studies will have included a sample size calculation by a suitably expert statistician to decide the number of children needed for the study. This is important since there is little point recruiting more children than needed as it would waste money and (if the intervention has no positive effect) could subject more children than necessary to a poor intervention. It may be rather harsh in areas where we are dealing with low incidence problems such as autism to downgrade the quality of evidence for an intervention using this (almost) arbitrary 400 children requirement.

It is also quite possible that many of the committee members (and I’m sure they’re not allowed to say!) may agree with some or all of the points that I make below. It is hard to understand though what latitude the guidelines development group has in making recommendations. Much of what they may have wanted to drawn upon isn’t available to be used as the basis for recommendations – because of the NICE review process. So, it will be interesting to see what happens with the final version of the autism NICE guidance.

So, to be clear – pretty much all of what I have to say below is not directed to the people who gave their time and expertise (including the lay members) to the committee. It is more about the overall process.

Autism as a medical disorder

One thing that stands out from the NICE document is that the report comes primarily from a perspective of autism as a medical disorder. I’ve written about this in a blog before (see http://profhastings.blogspot.co.uk/2013/03/recovery-from-autism-false-hope-or.html). There is a long and detailed chapter on interventions that aim to deal with the core symptoms of autism. Interesting in two respects. The first is the “aim” phrasing. Because the theoretical sensibleness of reviewed interventions is not included in the NICE format for reviewing (i.e., whether the intervention makes sense is not an inclusion criterion), then the use of “aim” is wrong. In practice, it instead means studies that happen to have included a measure of autism symptoms as an outcome measure. Forgive the example, but this means that a horseriding RCT was seriously evaluated despite no discussion of whether this made sense. Basically, the “how” question is not considered – how exactly is horseriding for 12 weeks once a week meant to be an intervention for the core symptoms of autism?

The second thing about interventions aimed at the core symptoms of autism is that this is rarely the point of many autism interventions. Especially those interventions used in educational contexts often focus on making a difference to children with autism in key domains that have an impact on the quality of life of them and their families. To be fair, one of the later chapters of the draft guidance includes interventions that “aim” to address associated features (including adaptive behaviour, IQ and the like) – or more properly, interventions that include these variables as outcome measures. However, it still makes me uncomfortable that curing symptoms of autism (although “cure” is of course not used as a term) gets such prominence.

Missing evidence

The data included to inform the guideline are based on outcomes of RCTs, but also qualitative research studies (a huge number of them in this draft guideline) and economic evidence. Although it is clear at several points that systematic reviews and meta-analyses would be within scope, I assume that this means only such analyses that are focused on RCTs alone. Otherwise, it is puzzling why a number of recent well-cited reviews are not discussed.

Economic evidence seems to be discussed with much less of an interest initially in its quality (although clearly quality of evidence weighs heavily in the conclusions). This does lead to the perverse situation on occasion of the discussion of economic analysis evidence for interventions that are not actually included in the outcome evidence review.

Qualitative research studies do not seem to have been identified for their quality, but for their relevance to review questions. There is also no requirement for qualitative studies to include a minimum number of participants. Despite this, the qualitative research evidence is used as the basis for a series of recommendations.

I am not necessarily criticising the inclusion of lower quality evidence or evidence where quality is not assessed in detail (although this does seem a bit weird!), but I do want to point out the perversity of this when a whole domain of evidence relating to autism is missing from the reviews informing the draft guidance. Specifically, Single Case Experimental Design (SCED, sometimes called Single Subject Experimental Design) studies have not been reviewed. These are not descriptions of single clinical cases, nor are they case series descriptions. Rather, they are a method used to bring a controlled analysis to studies involving a small number of participants. Strategies include multiple baselines (introducing an intervention to one child at a time, and showing that behaviour changes only when the intervention is introduced each time) and reversal designs (apply the intervention, remove it, and then reinstate to show the behaviour change is related to the intervention being in place).

Several other evidence review bodies throughout the world have recognised that SCED evidence is of value and needs to be reviewed when the effectiveness of an intervention is being addressed. Several bodies (including the What Works Clearing House) have established methods and criteria for reviewing SCED evidence and the level of evidence and replication needed to recommend an intervention as evidence-based.

Why does this matter? Well, a huge amount of the intervention research in the field of autism uses behavioural methods to teach skills or to reduce problematic behaviours. Single and focused behavioural interventions are often evaluated using SCED approaches. Thus, a whole bunch of relevant evidence is being excluded. Unless this issue is addressed by NICE it will have a serious impact on other upcoming reviews. Specifically, challenging behaviour in adults with learning disability is a topic in the early stages of the guideline development process. Again, many of the interventions use SCED evaluation methods. With the expansion of NICE to social care, there are also likely to be other reviews affected in future (for example, the same problem would occur if NICE reviewed supported employment interventions for adults with learning disability).

The current exclusion of SCED evidence is likely to be detrimental to people with disabilities. One might even argue that it is potentially discriminatory - since SCED evaluation methods have been primarily (though not exclusively) applied to interventions used with people with disabilities and not to (or less so with) other groups.

However, my main point for now is that given qualitative research evidence is accepted as a part of NICE evidence reviews then SCED evidence really ought to be too.

The development of evidence for complex interventions

The NICE draft autism guideline clearly states that interventions for autism are complex in nature. It is then surprising that guidelines about the development of evidence for complex interventions based on Medical Research Council and other frameworks (see my blog http://profhastings.blogspot.co.uk/2012/12/who-do-you-believe-model-for.html) do not seem to feature in the NICE review process. It is clear that various stages are needed to build evidence for complex interventions. At the very start, theoretical development is important (see earlier comment about horses!) – interventions have to make sense theoretically speaking rather than just being tried because they seem like a nice idea. This is especially a problem in autism it seems, and it is certainly good to have (in the draft NICE guidance) some “do not use” recommendations for interventions in autism that few scientists ever thought made proper sense.

Other stages of the process make it clear that different questions are asked at different stages. So, studies designed to test efficacy will look very different to those designed to test effectiveness. Similarly, studies designed to test large scale implementation may look different. RCT methods can be used at all of these stages but may not be, and it is probably reasonable to rely less on RCT designs as one gets towards the latter stages of building evidence for a complex intervention. There are several large scale implementation studies using controlled (but not RCT designs) that will have been excluded from the NICE review just because they were not RCTs.

The problem is that the inclusion criteria for evidence about intervention for autism is focused specifically on the use of RCT designs. No evaluation has been made about whether the study using a RCT design was actually designed to answer the question about an intervention’s effectiveness. I assume that this is a general issue for NICE reviews – perhaps the methodology has not yet caught up with the fact that NICE review a range of complex interventions and now also in a variety of settings including social care.

The problem of behavioural interventions

Many people are going to be disappointed or angry that the evidence for behavioural interventions for children with autism does not appear to have been reviewed as a part of the NICE process. I am going to offer a few thoughts on this since I know that: (a) some people might be interested, and (b) I can imagine the debate may get quite unpleasant.

First point comes back to the SCED evidence for autism interventions (see above). At present, this is clearly not evidence that can be included in the NICE process. It should be. However, it is difficult at this point to complain that it isn’t there. An implication for the NICE guideline though is that some statements will need to be re-worded. For example, it is unlikely to be true that there are no interventions specifically for addressing repetitive behaviours or rituals in children with autism. I suspect that there will be at least some SCED studies. Thus, it needs to be clearer that there are no studies meeting the restrictive NICE inclusion criteria. Also, when there is apparently no other evidence, you would think that taking a look at SCED evaluations would actually be rather useful as a way of developing recommendations.

A second point is that there could be more consistency in how behavioural approaches are described throughout the text of the guideline. The committee tentatively recommend the LEAP intervention model, for example. They are also clear that many interventions (including the Early Start Denver Model) use ABA methods. PECS is also discussed positively, and PECS is an ABA intervention of course. In fact, my reading of the LEAP evaluation paper that the committee liked so much is that it is also a behavioural intervention. Granted, discrete trial teaching methods are not the core of the approach, but all of the core elements of the intervention described by the study authors are ABA teaching technologies. There is also evaluation research from the UK where a variety of behavioural methods were used together as an educational approach in the early years and in a mainstream setting (like the LEAP model) [see Grindle, C., Hastings, R. P., Saville, M., Hughes, J. C., Kovshoff, H., & Huxley, K. (2009). Integrating evidence-based behavioural teaching methods into education for children with autism. Educational and Child Psychology, 26(4), 65-81. And Grindle, C. F., Hastings, R. P., Saville, M., Hughes, J. C., Huxley, K., Kovshoff, H., Griffith, G. M., Walker-Jones, E., Devonshire, K., & Remington, B. (2012). Outcomes of a behavioral education model for children with autism in a mainstream school setting. Behavior Modification, 36, 298-319.]

It would be good to see the guideline include a specific section that draws together the guideline development group’s perspective on behavioural methods. It is important to point out how many interventions that don’t even call themselves “ABA” or behavioural actually rely almost exclusively or in large part on ABA teaching technologies. The guideline group could help a rapprochement and future collaboration between autism professionals/experts to the benefit of children with autism and their families. I almost expected some discussion of this sort near the beginning of the draft guideline document where intervention “conceptual frameworks” are mentioned. However, this was instead a rather disappointing section.

So, this is a plea directly to the members of the guideline group – please, please, do something positive about behavioural interventions/methods along the lines I suggest. Otherwise the guidance is in serious danger of just reinforcing the fault lines that are already doing our field (and children with autism and their families) no good.

A final point on the behavioural stuff is that it is encouraging to see recommendations about the use of functional analysis and behavioural interventions based on functional analysis when dealing with challenging behaviours. The language used could do with a thorough edit though. For example, experts in behaviour analysis might reserve the term “functional analysis” for experimental functional analysis, which is a technique requiring considerable expertise and also ethical review. The term “functional assessment” may be what is meant. A small group of experts in ABA and Positive Behaviour Support would be able to take on an editing task. In addition, the guideline probably should clarify the competencies that may be needed to carry out some interventions (the example of experimental functional analysis is an example). The guideline could helpfully recommend referral to expert challenging behaviour teams or to a clearly (behaviourally) trained challenging behaviour expert within a multi-disciplinary team.