The most significant, and original, data on the outcomes of early intervention in autism?

This blog is focused on outcome data for Intensive Behavioural Intervention. I wanted to explain some of the findings that we generated a couple of years ago now drawing on data on the outcomes for individual children from a total of over 400 children with autism who had received early intervention based on a behavioural model or some education as usual comparison (or even another active intervention). The reason for this is that I think these data are probably the most important for policy and practice that have been published on behavioural intervention for autism. Of course I am biased. However, I have also been reflecting on which research papers get picked up by other scientists and practitioners/policy makers. How come data that I think are some of the most interesting we’ve ever produced seem to go relatively ignored?

Meta- and Mega-Analysis of intervention outcome data

Individual research evaluation studies of interventions are important and have their place in telling an aspect of the evidence story (see blog: http://profhastings.blogspot.co.uk/2012/12/using-evidence-tool-to-draw-your-own.html ). At some point, data across several studies can be combined in what is known as meta-analysis. The results of at least two studies of the same intervention, and using the same outcome measures, are summarised to give hopefully a less biased idea of the effect of an intervention’s effects. There is also a notion that accumulating evidence over several studies is more convincing that individual studies alone. So, meta-analysis can be used to summarise accumulating evidence for an intervention approach.

In meta-analysis, the data analysis focuses at the level of the individual studies. Although there are several complexities to the summary process (e.g., giving a stronger weighting to the evidence from higher quality studies) what essentially happens is that the meta-analysis results focus on the average effect of the intervention across several studies.

We have published a meta-analysis summarising the results of Intensive Behavioural Intervention (i.e., ABA intervention) from nine research studies (http://www.tandfonline.com/doi/abs/10.1080/15374410902851739). The results, like those of several other meta-analyses of behavioural intervention, were positive.

A more unique step is to focus an analysis across several published studies at the level of the individual children in those studies. To do this, a process called mega-analysis (or Individual Participant Data Meta-Analysis – I like “mega-analysis” better!) can be used. What you do here is, like with a meta-analysis, start by systematically indentifying all studies that meet your inclusion criteria for the review (i.e., are actually focused on evaluation of the intervention in which you are interested). You then write to the researchers who conducted each study and ask for the data on all of the children they reported on in their research paper. Once you have collected all of these data, you can carry out an analysis as if all the children were a part of a single very large study (i.e., a large group of children who have received the intervention, and also a large group of children who received education as usual or a comparison intervention). One advantage of mega-analysis is that you can increase the number of studies you include for analysis because those simply focused on the evaluation of an intervention group only (where no control or comparison group was originally recruited) can be included in the overall “big” intervention group.

Our 2010 Mega-Analysis

Using a systematic review process, we identified the same nine controlled studies included in our earlier more traditional meta-analysis plus a further seven studies that did not include a suitable control condition. In total, these studies had reported data on 309 children who received behavioural intervention and 144 who received education as usual or a comparison intervention. You can see information about the study including its abstract (summary) at the following URL: http://www.aaiddjournals.org/doi/abs/10.1352/1944-7558-115.5.381

The researchers of all of the studies we identified gave us their data on the individual children (anonymised of course) – a 100% return rate to our request. Researchers who responded also helpfully identified a case where a small number of children had been included in more than one outcome study so we were able to make sure that these children featured only once in our analysis. This is something that would be impossible to do in a meta-analysis study.

To reflect the focus on individual children’s outcomes, we adopted an analysis approach that assesses the extent of change seen at an individual level. The method we used was the Reliable Change Index (http://psycnet.apa.org/journals/ccp/59/1/12/ ). What happens here is that a statistical formula is used to identify how much an individual child’s score on an outcome measure needs to change by for us to be 95% sure that the change is a meaningful change and not just down to natural measurement variation or the reliability of the measurement tool. So, in the end we got a Yes/No outcome for each child – did their score on a key measure change by this identified amount, or not. The detail isn’t important, but we focused on two outcomes in our mega-analysis: change in IQ test scores, and also change in a standardised measure of adaptive behaviour skills called the Vineland Adaptive Behavior Scales (communication skills, social skills, and daily living skills). The statistical formula showed that to meet the criteria for reliable change at the level of the individual child, IQ needed to change by more than 27 standard points over two years of intervention, and adaptive skills by at least 20 standard points over two years of intervention.

We used both IQ and adaptive behaviour “standard scores”. You will be familiar with this idea probably from IQ where if someone’s IQ is 100, they have scored on an IQ test at the level expected for someone of their age. So, over time, standard scores normally change little – since individuals are likely to learn/develop at a standard rate. If someone performs at an average level at one age, they are likely to perform at an average age also at a later age. So, their IQ (for example) is likely to stay around 100 for a long period of time. Increases in standard scores over time suggest that the individual has learned new things at a faster rate/developed faster than would have been expected for someone of their age at the point they started from. So, having to increase your scores on a standardised test by 20-27 points over two years is quite a decent amount and perhaps quite a tough criterion.

Our results are summarised in the graph below:

What this graphs shows is that a much larger % of the 309 children in the behavioural intervention group changed to a reliable extent on IQ and adaptive behaviour over two years compared to the 144 children in the control/comparison intervention group. This is, as far as I am aware, the largest controlled comparison of outcomes from behavioural intervention for young children with autism ever reported.

Why is this study important, and unique?

So, I have already suggested that our 2010 study is the biggest analysis of outcomes from comprehensive behavioural intervention for young children with autism ever published. As far as I can tell, it is also perhaps the only (or one of a very small number of) mega-analysis ever conducted in the field of autism. More broadly, mega-analyses typically contain only a proportion of the individual data from the studies identified for review (because researchers just don’t let other people have access to their data, or make it awkward so it doesn’t happen). So, this study may well be pretty much unique in the field of intellectual and developmental disability and special education for achieving a 100% response to the request for individual child data. In all of these ways, our study may be unique or at the very least a rare example of the successful application of a mega-analysis approach.

The data from the meta-analysis are also important for the unique perspective that they can provide for policy makers and in terms of an economic analysis of comprehensive behavioural intervention for young children with autism. Basically, we showed that around 30% of children with autism receiving a comprehensive behavioural intervention might have very considerable positive outcomes after two years of intervention and that children who do not receive this intervention are much less likely to achieve this level of outcome (about 7% at best). Thus, behavioural intervention might increase the chances of a considerably positive outcome by 4-5 times. These data might be a useful way for policy makers to consider the potential impact of large scale implementation of a comprehensive behavioural intervention model and also to begin ask whether the economic investment (quite apart from the value of positive outcomes for an individual child and their family) is worthwhile.

A final reflection for this blog is maybe most interesting for other researchers. I hope that I have shown that these data are hugely important, and perhaps unique in the field of autism. One metric of whether other people recognise the significance of a piece of research is whether it gets cited in other publications in the scientific literature (i.e., if someone else includes a reference to the research when they publish their results or review some evidence). Our META-analysis (Eldevik et al. 2009) has 134 citations recorded in Google Scholar as of today, whereas the MEGA-analysis (Eldevik et al., 2010) has 20 citations. Why aren’t people apparently picking up on the significance of the mega-analysis data we published? Am I just wrong about how potentially important and interesting the results are? Or have we just failed to communicate to people what these data mean? Any thoughts gratefully received!!