The most significant, and original, data on the outcomes of early intervention in autism?

This blog is focused on outcome data for Intensive Behavioural Intervention. I wanted to explain some of the findings that we generated a couple of years ago now drawing on data on the outcomes for individual children from a total of over 400 children with autism who had received early intervention based on a behavioural model or some education as usual comparison (or even another active intervention). The reason for this is that I think these data are probably the most important for policy and practice that have been published on behavioural intervention for autism. Of course I am biased. However, I have also been reflecting on which research papers get picked up by other scientists and practitioners/policy makers. How come data that I think are some of the most interesting we’ve ever produced seem to go relatively ignored?

Meta- and Mega-Analysis of intervention outcome data

Individual research evaluation studies of interventions are important and have their place in telling an aspect of the evidence story (see blog: http://profhastings.blogspot.co.uk/2012/12/using-evidence-tool-to-draw-your-own.html ). At some point, data across several studies can be combined in what is known as meta-analysis. The results of at least two studies of the same intervention, and using the same outcome measures, are summarised to give hopefully a less biased idea of the effect of an intervention’s effects. There is also a notion that accumulating evidence over several studies is more convincing that individual studies alone. So, meta-analysis can be used to summarise accumulating evidence for an intervention approach.

In meta-analysis, the data analysis focuses at the level of the individual studies. Although there are several complexities to the summary process (e.g., giving a stronger weighting to the evidence from higher quality studies) what essentially happens is that the meta-analysis results focus on the average effect of the intervention across several studies.

We have published a meta-analysis summarising the results of Intensive Behavioural Intervention (i.e., ABA intervention) from nine research studies (http://www.tandfonline.com/doi/abs/10.1080/15374410902851739). The results, like those of several other meta-analyses of behavioural intervention, were positive.

A more unique step is to focus an analysis across several published studies at the level of the individual children in those studies. To do this, a process called mega-analysis (or Individual Participant Data Meta-Analysis – I like “mega-analysis” better!) can be used. What you do here is, like with a meta-analysis, start by systematically indentifying all studies that meet your inclusion criteria for the review (i.e., are actually focused on evaluation of the intervention in which you are interested). You then write to the researchers who conducted each study and ask for the data on all of the children they reported on in their research paper. Once you have collected all of these data, you can carry out an analysis as if all the children were a part of a single very large study (i.e., a large group of children who have received the intervention, and also a large group of children who received education as usual or a comparison intervention). One advantage of mega-analysis is that you can increase the number of studies you include for analysis because those simply focused on the evaluation of an intervention group only (where no control or comparison group was originally recruited) can be included in the overall “big” intervention group.

Our 2010 Mega-Analysis

Using a systematic review process, we identified the same nine controlled studies included in our earlier more traditional meta-analysis plus a further seven studies that did not include a suitable control condition. In total, these studies had reported data on 309 children who received behavioural intervention and 144 who received education as usual or a comparison intervention. You can see information about the study including its abstract (summary) at the following URL: http://www.aaiddjournals.org/doi/abs/10.1352/1944-7558-115.5.381

The researchers of all of the studies we identified gave us their data on the individual children (anonymised of course) – a 100% return rate to our request. Researchers who responded also helpfully identified a case where a small number of children had been included in more than one outcome study so we were able to make sure that these children featured only once in our analysis. This is something that would be impossible to do in a meta-analysis study.

To reflect the focus on individual children’s outcomes, we adopted an analysis approach that assesses the extent of change seen at an individual level. The method we used was the Reliable Change Index (http://psycnet.apa.org/journals/ccp/59/1/12/ ). What happens here is that a statistical formula is used to identify how much an individual child’s score on an outcome measure needs to change by for us to be 95% sure that the change is a meaningful change and not just down to natural measurement variation or the reliability of the measurement tool. So, in the end we got a Yes/No outcome for each child – did their score on a key measure change by this identified amount, or not. The detail isn’t important, but we focused on two outcomes in our mega-analysis: change in IQ test scores, and also change in a standardised measure of adaptive behaviour skills called the Vineland Adaptive Behavior Scales (communication skills, social skills, and daily living skills). The statistical formula showed that to meet the criteria for reliable change at the level of the individual child, IQ needed to change by more than 27 standard points over two years of intervention, and adaptive skills by at least 20 standard points over two years of intervention.

We used both IQ and adaptive behaviour “standard scores”. You will be familiar with this idea probably from IQ where if someone’s IQ is 100, they have scored on an IQ test at the level expected for someone of their age. So, over time, standard scores normally change little – since individuals are likely to learn/develop at a standard rate. If someone performs at an average level at one age, they are likely to perform at an average age also at a later age. So, their IQ (for example) is likely to stay around 100 for a long period of time. Increases in standard scores over time suggest that the individual has learned new things at a faster rate/developed faster than would have been expected for someone of their age at the point they started from. So, having to increase your scores on a standardised test by 20-27 points over two years is quite a decent amount and perhaps quite a tough criterion.

Our results are summarised in the graph below:

What this graphs shows is that a much larger % of the 309 children in the behavioural intervention group changed to a reliable extent on IQ and adaptive behaviour over two years compared to the 144 children in the control/comparison intervention group. This is, as far as I am aware, the largest controlled comparison of outcomes from behavioural intervention for young children with autism ever reported.

Why is this study important, and unique?

So, I have already suggested that our 2010 study is the biggest analysis of outcomes from comprehensive behavioural intervention for young children with autism ever published. As far as I can tell, it is also perhaps the only (or one of a very small number of) mega-analysis ever conducted in the field of autism. More broadly, mega-analyses typically contain only a proportion of the individual data from the studies identified for review (because researchers just don’t let other people have access to their data, or make it awkward so it doesn’t happen). So, this study may well be pretty much unique in the field of intellectual and developmental disability and special education for achieving a 100% response to the request for individual child data. In all of these ways, our study may be unique or at the very least a rare example of the successful application of a mega-analysis approach.

The data from the meta-analysis are also important for the unique perspective that they can provide for policy makers and in terms of an economic analysis of comprehensive behavioural intervention for young children with autism. Basically, we showed that around 30% of children with autism receiving a comprehensive behavioural intervention might have very considerable positive outcomes after two years of intervention and that children who do not receive this intervention are much less likely to achieve this level of outcome (about 7% at best). Thus, behavioural intervention might increase the chances of a considerably positive outcome by 4-5 times. These data might be a useful way for policy makers to consider the potential impact of large scale implementation of a comprehensive behavioural intervention model and also to begin ask whether the economic investment (quite apart from the value of positive outcomes for an individual child and their family) is worthwhile.

A final reflection for this blog is maybe most interesting for other researchers. I hope that I have shown that these data are hugely important, and perhaps unique in the field of autism. One metric of whether other people recognise the significance of a piece of research is whether it gets cited in other publications in the scientific literature (i.e., if someone else includes a reference to the research when they publish their results or review some evidence). Our META-analysis (Eldevik et al. 2009) has 134 citations recorded in Google Scholar as of today, whereas the MEGA-analysis (Eldevik et al., 2010) has 20 citations. Why aren’t people apparently picking up on the significance of the mega-analysis data we published? Am I just wrong about how potentially important and interesting the results are? Or have we just failed to communicate to people what these data mean? Any thoughts gratefully received!!

Autism Spectrum Disorder Mental Health Blog

It is clear that Mental Health is a big issue for children, adolescents and adults with autism. People with autism appear to be at considerable increased risk for mental health problems. Despite the significance of the issue for both people with autism and their families, high quality research is rare.

If you're interested in topics relating to mental health in children and adolescents with autism and, crucially, how to support carers, then check out this new blog from friend and colleague Dr Jonathan Weiss from York University in Toronto Canada. Jonathan and his students are posting useful blogs and research summaries. Worth also keeping an eye out for new findings that will come from Jonathan's new research programme on these topics.

http://asdmentalhealth.blog.yorku.ca/

Autism and Evidence 6. What is wrong with the draft NICE guidance on “management” for children with autism?

What is wrong with the draft NICE guidance on “management” for children with autism?

I promised to post something on this blog about my thoughts on the NICE clinical guideline for autism in children currently out for consultation (http://www.nice.org.uk/guidance/index.jsp?action=folder&o=63399). This blog isn’t in the form of the response that I might submit to NICE partly because they ask for a response using a standard form rather than a text story. I’m also not going to write about details here – more ideas and broader issues.

As a starting point, I’d like to acknowledge how tough it must have been for the experts and other members of the committee who worked on this NICE guideline. It is pretty clear in the document that NICE procedures were rigorously (and rigidly) applied. This includes the use of the GRADE system that surely was not designed for the purpose of understanding the evidence for complex interventions in contexts that will not be the same as short primary or secondary healthcare consultation. For example, the rating of quality of evidence from any intervention included in this NICE autism review has been downgraded if the meta-analysis sample is under 400 children (because any smaller sample may not provide a good estimate of the size of effect of an intervention). This is despite some well-designed individual trials that passed strict peer review for competitive funding and were published in some of the highest impact scientific journals. Such studies will have included a sample size calculation by a suitably expert statistician to decide the number of children needed for the study. This is important since there is little point recruiting more children than needed as it would waste money and (if the intervention has no positive effect) could subject more children than necessary to a poor intervention. It may be rather harsh in areas where we are dealing with low incidence problems such as autism to downgrade the quality of evidence for an intervention using this (almost) arbitrary 400 children requirement.

It is also quite possible that many of the committee members (and I’m sure they’re not allowed to say!) may agree with some or all of the points that I make below. It is hard to understand though what latitude the guidelines development group has in making recommendations. Much of what they may have wanted to drawn upon isn’t available to be used as the basis for recommendations – because of the NICE review process. So, it will be interesting to see what happens with the final version of the autism NICE guidance.

So, to be clear – pretty much all of what I have to say below is not directed to the people who gave their time and expertise (including the lay members) to the committee. It is more about the overall process.

Autism as a medical disorder

One thing that stands out from the NICE document is that the report comes primarily from a perspective of autism as a medical disorder. I’ve written about this in a blog before (see http://profhastings.blogspot.co.uk/2013/03/recovery-from-autism-false-hope-or.html). There is a long and detailed chapter on interventions that aim to deal with the core symptoms of autism. Interesting in two respects. The first is the “aim” phrasing. Because the theoretical sensibleness of reviewed interventions is not included in the NICE format for reviewing (i.e., whether the intervention makes sense is not an inclusion criterion), then the use of “aim” is wrong. In practice, it instead means studies that happen to have included a measure of autism symptoms as an outcome measure. Forgive the example, but this means that a horseriding RCT was seriously evaluated despite no discussion of whether this made sense. Basically, the “how” question is not considered – how exactly is horseriding for 12 weeks once a week meant to be an intervention for the core symptoms of autism?

The second thing about interventions aimed at the core symptoms of autism is that this is rarely the point of many autism interventions. Especially those interventions used in educational contexts often focus on making a difference to children with autism in key domains that have an impact on the quality of life of them and their families. To be fair, one of the later chapters of the draft guidance includes interventions that “aim” to address associated features (including adaptive behaviour, IQ and the like) – or more properly, interventions that include these variables as outcome measures. However, it still makes me uncomfortable that curing symptoms of autism (although “cure” is of course not used as a term) gets such prominence.

Missing evidence

The data included to inform the guideline are based on outcomes of RCTs, but also qualitative research studies (a huge number of them in this draft guideline) and economic evidence. Although it is clear at several points that systematic reviews and meta-analyses would be within scope, I assume that this means only such analyses that are focused on RCTs alone. Otherwise, it is puzzling why a number of recent well-cited reviews are not discussed.

Economic evidence seems to be discussed with much less of an interest initially in its quality (although clearly quality of evidence weighs heavily in the conclusions). This does lead to the perverse situation on occasion of the discussion of economic analysis evidence for interventions that are not actually included in the outcome evidence review.

Qualitative research studies do not seem to have been identified for their quality, but for their relevance to review questions. There is also no requirement for qualitative studies to include a minimum number of participants. Despite this, the qualitative research evidence is used as the basis for a series of recommendations.

I am not necessarily criticising the inclusion of lower quality evidence or evidence where quality is not assessed in detail (although this does seem a bit weird!), but I do want to point out the perversity of this when a whole domain of evidence relating to autism is missing from the reviews informing the draft guidance. Specifically, Single Case Experimental Design (SCED, sometimes called Single Subject Experimental Design) studies have not been reviewed. These are not descriptions of single clinical cases, nor are they case series descriptions. Rather, they are a method used to bring a controlled analysis to studies involving a small number of participants. Strategies include multiple baselines (introducing an intervention to one child at a time, and showing that behaviour changes only when the intervention is introduced each time) and reversal designs (apply the intervention, remove it, and then reinstate to show the behaviour change is related to the intervention being in place).

Several other evidence review bodies throughout the world have recognised that SCED evidence is of value and needs to be reviewed when the effectiveness of an intervention is being addressed. Several bodies (including the What Works Clearing House) have established methods and criteria for reviewing SCED evidence and the level of evidence and replication needed to recommend an intervention as evidence-based.

Why does this matter? Well, a huge amount of the intervention research in the field of autism uses behavioural methods to teach skills or to reduce problematic behaviours. Single and focused behavioural interventions are often evaluated using SCED approaches. Thus, a whole bunch of relevant evidence is being excluded. Unless this issue is addressed by NICE it will have a serious impact on other upcoming reviews. Specifically, challenging behaviour in adults with learning disability is a topic in the early stages of the guideline development process. Again, many of the interventions use SCED evaluation methods. With the expansion of NICE to social care, there are also likely to be other reviews affected in future (for example, the same problem would occur if NICE reviewed supported employment interventions for adults with learning disability).

The current exclusion of SCED evidence is likely to be detrimental to people with disabilities. One might even argue that it is potentially discriminatory - since SCED evaluation methods have been primarily (though not exclusively) applied to interventions used with people with disabilities and not to (or less so with) other groups.

However, my main point for now is that given qualitative research evidence is accepted as a part of NICE evidence reviews then SCED evidence really ought to be too.

The development of evidence for complex interventions

The NICE draft autism guideline clearly states that interventions for autism are complex in nature. It is then surprising that guidelines about the development of evidence for complex interventions based on Medical Research Council and other frameworks (see my blog http://profhastings.blogspot.co.uk/2012/12/who-do-you-believe-model-for.html) do not seem to feature in the NICE review process. It is clear that various stages are needed to build evidence for complex interventions. At the very start, theoretical development is important (see earlier comment about horses!) – interventions have to make sense theoretically speaking rather than just being tried because they seem like a nice idea. This is especially a problem in autism it seems, and it is certainly good to have (in the draft NICE guidance) some “do not use” recommendations for interventions in autism that few scientists ever thought made proper sense.

Other stages of the process make it clear that different questions are asked at different stages. So, studies designed to test efficacy will look very different to those designed to test effectiveness. Similarly, studies designed to test large scale implementation may look different. RCT methods can be used at all of these stages but may not be, and it is probably reasonable to rely less on RCT designs as one gets towards the latter stages of building evidence for a complex intervention. There are several large scale implementation studies using controlled (but not RCT designs) that will have been excluded from the NICE review just because they were not RCTs.

The problem is that the inclusion criteria for evidence about intervention for autism is focused specifically on the use of RCT designs. No evaluation has been made about whether the study using a RCT design was actually designed to answer the question about an intervention’s effectiveness. I assume that this is a general issue for NICE reviews – perhaps the methodology has not yet caught up with the fact that NICE review a range of complex interventions and now also in a variety of settings including social care.

The problem of behavioural interventions

Many people are going to be disappointed or angry that the evidence for behavioural interventions for children with autism does not appear to have been reviewed as a part of the NICE process. I am going to offer a few thoughts on this since I know that: (a) some people might be interested, and (b) I can imagine the debate may get quite unpleasant.

First point comes back to the SCED evidence for autism interventions (see above). At present, this is clearly not evidence that can be included in the NICE process. It should be. However, it is difficult at this point to complain that it isn’t there. An implication for the NICE guideline though is that some statements will need to be re-worded. For example, it is unlikely to be true that there are no interventions specifically for addressing repetitive behaviours or rituals in children with autism. I suspect that there will be at least some SCED studies. Thus, it needs to be clearer that there are no studies meeting the restrictive NICE inclusion criteria. Also, when there is apparently no other evidence, you would think that taking a look at SCED evaluations would actually be rather useful as a way of developing recommendations.

A second point is that there could be more consistency in how behavioural approaches are described throughout the text of the guideline. The committee tentatively recommend the LEAP intervention model, for example. They are also clear that many interventions (including the Early Start Denver Model) use ABA methods. PECS is also discussed positively, and PECS is an ABA intervention of course. In fact, my reading of the LEAP evaluation paper that the committee liked so much is that it is also a behavioural intervention. Granted, discrete trial teaching methods are not the core of the approach, but all of the core elements of the intervention described by the study authors are ABA teaching technologies. There is also evaluation research from the UK where a variety of behavioural methods were used together as an educational approach in the early years and in a mainstream setting (like the LEAP model) [see Grindle, C., Hastings, R. P., Saville, M., Hughes, J. C., Kovshoff, H., & Huxley, K. (2009). Integrating evidence-based behavioural teaching methods into education for children with autism. Educational and Child Psychology, 26(4), 65-81. And Grindle, C. F., Hastings, R. P., Saville, M., Hughes, J. C., Huxley, K., Kovshoff, H., Griffith, G. M., Walker-Jones, E., Devonshire, K., & Remington, B. (2012). Outcomes of a behavioral education model for children with autism in a mainstream school setting. Behavior Modification, 36, 298-319.]

It would be good to see the guideline include a specific section that draws together the guideline development group’s perspective on behavioural methods. It is important to point out how many interventions that don’t even call themselves “ABA” or behavioural actually rely almost exclusively or in large part on ABA teaching technologies. The guideline group could help a rapprochement and future collaboration between autism professionals/experts to the benefit of children with autism and their families. I almost expected some discussion of this sort near the beginning of the draft guideline document where intervention “conceptual frameworks” are mentioned. However, this was instead a rather disappointing section.

So, this is a plea directly to the members of the guideline group – please, please, do something positive about behavioural interventions/methods along the lines I suggest. Otherwise the guidance is in serious danger of just reinforcing the fault lines that are already doing our field (and children with autism and their families) no good.

A final point on the behavioural stuff is that it is encouraging to see recommendations about the use of functional analysis and behavioural interventions based on functional analysis when dealing with challenging behaviours. The language used could do with a thorough edit though. For example, experts in behaviour analysis might reserve the term “functional analysis” for experimental functional analysis, which is a technique requiring considerable expertise and also ethical review. The term “functional assessment” may be what is meant. A small group of experts in ABA and Positive Behaviour Support would be able to take on an editing task. In addition, the guideline probably should clarify the competencies that may be needed to carry out some interventions (the example of experimental functional analysis is an example). The guideline could helpfully recommend referral to expert challenging behaviour teams or to a clearly (behaviourally) trained challenging behaviour expert within a multi-disciplinary team.