Who's Challenging Who - example of good practice in Winterbourne View final report

Today the UK government Department of Health published its final report on the Winterbourne View care scandal. A joint project between Bangor University and Mencap ("Who's Challenging Who") is described in the Good Practice Examples accompanying report. The Good Practice report, along with the full review report and Easy Read versions can be downloaded for free at:

http://www.dh.gov.uk/health/2012/12/final-winterbourne/

The summary of the research evaluation of Who's Challenging Who is now in press with the Journal of Intellectual Disability Research:

http://onlinelibrary.wiley.com/doi/10.1111/j.1365-2788.2012.01630.x/abstract

The Department of Health Good Practice Examples report includes the following information about Who's Challenging Who (quoted from page 80 of the report):

"The Who’s Challenging Who project

Who’s Challenging Who (WCW) is an attitude change training package that has been developed over 18 months since January 2011 (pre‐dating Winterbourne View). The development and pilot evaluation of WCW was funded by a Knowledge Transfer Partnership grant between Mencap and Bangor University.

WCW involves a person with learning disability who has behaviour that challenges working with a person without disability as co‐trainers for a half day workshop targeting health and social care support staff.

The WCW training curriculum was informed jointly by the experiences of the co-trainers with learning disability and by existing research evidence. The theory behind the WCW training is to provide multiple opportunities for direct contact with individuals whose behaviour challenges and contact with information about their experiences of services and being labelled as “challenging”. Small groups of support staff (up to 10) experience an intensive interactive session with the aim of increasing their empathy (what it is like to be in the shoes of individuals labelled as challenging), and changing their attitudes. An outline of Who’s Challenging Who 2 Contact theory (the theory behind WCW) has already been used successfully to inform attitude change and reduce stigma towards other excluded groups, such as individuals with mental health difficulties.

For more information - http://www.mencap.org.uk/wales/projects/who-s-challenging-who"

Autism and Evidence 4. Does ABA work for children with autism?

 This blog must be read after blogs 1-3 in this series where I describe a way of thinking about “evidence”, and also where I have explained what I think ABA is when applied to working in educational contexts with children with autism. I am assuming that you have read these - they are the context for the blog. Please don’t read the current blog out of context!

The first thing to point out is that the form of my question is just plain wrong. ABA is an applied science. So asking if ABA “works” is like asking if Physics works or Ecological Science works. We have to get more specific. The question needs to be posed about particular ABA-based interventions. So, it is more appropriate to ask whether PECS works for children with autism. It is also appropriate to ask whether Positive Behaviour Support works in reducing behaviour problems in children with autism. Thinking about some of our recent research, we have asked whether an Internet reading programme based on ABA principles works for children with autism, and whether using a buzzer prompting devise in a child’s pocket can help to improve their social initiations towards peers.

In this blog, I do not intend to review everything. For the moment, I will focus on whether a comprehensive ABA-based educational model leads to good outcomes for children with autism. So, my question might be about the evidence for Early Behavioural Intervention (whether Intensive or not).

A little more about my perspective

I also want to be clear about some of the reasons I am writing this particular blog (and the series). I am not a behaviour analyst. I am an academic psychologist and researcher working in the field of intellectual disability (learning disability using the UK health terminology) and autism. I have studied behaviour analysis and carried out a good deal of research on ABA-based interventions for children with autism. However, I am no spokesperson for ABA. I do hope that I have a contribution to make as someone who has some understanding of ABA but also more broadly psychology and perhaps policy/practice in the field of ID and autism.

It is important to say that I am convinced that there is something important in ABA interventions to support children with autism. I am frustrated both by the “pro” and the “anti” ABA camps out there. My opinion is that those pro ABA in the field of autism have been poor at communicating their work and also sometimes poor at working with other autism professionals and experts, including those who are autistic. I have to perhaps include criticism of myself within this too. I am trying hard though, and this blog is a part of that effort. Those anti ABA, in my opinion, have not taken the time to fully inform themselves about ABA, fail to recognize the need to specify the question about what ABA-based interventions might work, and are preventing children and young people with autism benefiting from something useful.

What am I “pro” then? I’m pro making a positive difference to the lives of children and young people with autism. I suspect anyone reading this blog will say the same thing. So, for goodness sake let’s talk and collaborate and stop getting sidetracked away from the most important agenda. Many children out there with autism even in the UK still receive no specialized support, people do not understand their condition, and when children receive some services there is poor practice (ABA and non-ABA alike). Children and young people also face that cliff at transition to adult services. The cliff is partly about even fewer services for adults, but also a real lack of continuity from an educational focus to…err…what?

The evidence for comprehensive Early Behavioural Intervention - a summary

My summary of the evidence can be represented by this single picture that only makes sense in the context of my earlier blogs:

Let me explain. A comprehensive ABA-based educational model for young children with autism (early - so perhaps up to around 7/8 years of age) is informed by basic scientific findings about learning processes, and also by research suggesting that children with autism can fail to develop typically during the early years. Comprehensive ABA models are also informed by research describing how children develop typically, and by autism research showing what skills early on might predict later positive outcomes (e.g., joint attention and other social skills, communication skills). These various aspects of scientific research have informed clearly described comprehensive long-term educational programmes much of which is described in manuals for interventionists. These interventions are conceptually rooted in behavioural theory.

There are plenty of studies exploring outcomes from the delivery of these described interventions, some of which include comparison or control conditions. Lovaas’ (1987) seminal study might be viewed as a part of this effort. Thus, Phases 0 and 1 in the evidence continuum are covered pretty well.

I am not aware of a definitive randomized controlled trial study yet published that was designed as a test of the efficacy of comprehensive early behavioural intervention. That is why I have a red cross at Phase 2 of the evidence continuum. Why is this? It is probably safest to say that no-one really knows. There may be many reasons. Historically, for example, when behaviour analysts were first delivering and evaluating comprehensive ABA-based intervention for children with autism (actually 30 years ago now or more) there may have been less clarity about the importance of efficacy RCTs. Another problem has always been what the comparison should be. As I pointed out in my third Blog in this series, ABA-based methods are probably being used as a part of pretty much all educational practice in the field of autism. So, how different might best practice and comprehensive ABA actually look?

This touches on an ethical point some researchers have made - that random allocation to an intervention lasting two years or more is ethically complex when one already has some evidence that a comprehensive ABA-based model may be beneficial. The control group could not benefit from early intervention once that period has passed for them. So, that possibility in their life would be lost. Others argue that RCTs of longer and more intensive interventions in the field of autism can indeed be carried out. There is no disagreement that this is possible (see Dawson Rogers et al ESDM study mentioned in my second Blog in this series, for an example) - that RCTs CAN be carried out. However, there is still an ethical debate to be had.

To be clear, the fact is that I am pretty sure there is no knock-out efficacy RCT of comprehensive ABA-based educational intervention for young children with autism. Commentators who point this out are absolutely correct. We do not disagree. However, the point in describing the evidence continuum to you is that we know this is not the whole story. Anyone stopping at this point without providing the full picture is doing a poor job of informing people about the evidence. The lack of an efficacy RCT is perhaps a gaping hole, but it does not invalidate the remainder of the evidence.

So, the full picture is that there are quite a lot of Phase 3 outcome studies relating to comprehensive ABA-based educational models. There are several effectiveness studies, including some use of randomized designs. For example, Smith, Groen and Wynn in 2000 published a paper where they had randomly allocated children to a comprehensive ABA-based early intervention which was quite intensive and also delivered by experts, or to an intervention where parents were trained in similar ABA skills (and they delivered fewer intervention hours). The children in the expert delivery group had better outcomes. There were still some positive outcomes for the parent training group too. What I would take from this study is that these may be two different delivery models for early behavioural comprehensive intervention that can be clearly described. Both may be viable models depending on resources available and family capacity, but if the studies results are replicated we might conclude that an expert delivered model is likely to be more effective. This sort of research could inform how intervention could be delivered on a large scale.

Comprehensive Early Behavioural Intervention - implementation research

What is pretty much unique in the field of autism interventions is that there are also data at the implementation end of the evidence continuum for ABA-based comprehensive early intervention. Several groups in Canada, for example, have delivered large scale (i.e., hundreds of children with autism) early behavioural intervention models and also evaluated outcomes and published the results. Large scale delivery is not simple, and several problems needed to be solved along the way including how to train enough interventionists in the first place, and how to ensure that services were delivering good quality intervention across large geographic areas. The outcome studies are very encouraging in terms of making a difference to the skills of young children with autism. Google (other search engines are available!) the work of Adrienne Perry, Nancy Freeman and colleagues in Ontario, and Isabel Smith, Susan Bryson and colleagues in Nova Scotia.

We have also recently published two implementation studies with colleagues in Wales [Grindle, C. F., Hastings, R. P., Saville, M., Hughes, J. C., Huxley, K., Kovshoff, H., Griffith, G. M., Walker-Jones, E., Devonshire, K., & Remington, B. (2012). Outcomes of a behavioral education model for children with autism in a
mainstream school setting. Behavior Modification, 36, 298-319], and in Norway [Eldevik, S., Hastings, R. P., Jahr, E., & Hughes, J. C. (2012). Outcomes of behavioral intervention for children with Autism in mainstream pre-school settings. Journal of Autism and Developmental Disorders, 42, 210-220]. In both studies, we described how ABA comprehensive models can be delivered within existing services (mainstream pre-schools, and a special class within a mainstream infant school), and showed that outcomes were better than existing “education as usual” models.

The main point here is that it would be a mistake to evaluate these research studies as if they were designed to test the efficacy of comprehensive ABA-based education for children with autism. They should properly be considered as Phase 4 research - developing/describing and evaluating the day-to-day delivery of comprehensive ABA intervention methods.

More implementation research (practice-based evidence) needed

We cannot wait 20+ years for basic science to inform new interventions and for these to go through the process of gathering evidence right to the implementation stage. More evidence is needed NOW to inform the design and development of educational services for children with autism. It would also be premature to base these service design and delivery discussions on evidence about interventions that have only reached the stage of being tested for their efficacy (Phase 2). Such interventions are very encouraging in terms of their outcomes, but we have no idea whether they will prove effective and whether they will deliver good outcomes on a large scale.

Intriguingly, ABA-based intervention, especially for young children with autism has been around so long that there are examples and data around about successful implementation. Service providers and commissioners must take notice of these unique data partly for what they say about ABA-based interventions but also the lessons that might be learned in relation to the delivery of other interventions.

Just as I advocated earlier in this blog for pro and anti ABA camps to work together towards a common goal, researchers and service providers need also to collaborate on implementation research. This need is urgent. Educational services and professionals need to clearly define their models and identify the evidence informing what they are trying to do. What is the rationale for your educational model? Researchers then need to turn to more applied research projects, devising ways to evaluate these defined models. We can then start to have evidence-based educational service delivery in the field of autism.

Funders of autism research can also play a role here. They all seem to recognize that there has been too much emphasis on funding basic science on autism and less attention given to intervention. However, it is important to use the evidence continuum model to inform funding of good quality research at all Phases. A focus only on Phase 2 efficacy trials will take a long time to have impact on children with autism, their families, and educators. Yes, children with autism and their families deserve the highest quality intervention research. However, what is the point if this work does not or cannot inform practice and services?

Autism Evidence 3. What is ABA for children with autism?

 Many people seem to misunderstand what Applied Behavioural Analysis (ABA) actually is when applied to the education of children with autism. I have found some limitations in understanding also among ABA folks themselves. This blog is the result of many discussions with colleagues about how to answer the question “What is ABA?” when applied to working with children with autism. However, the description presented here is my own perspective. Not all of my points will be agreed upon by all experts in ABA. See whether you find this blog helpful.

What is ABA?

ABA has evolved from a science of learning (called behaviour analysis). Basic scientific research has identified how humans learn, and the application of this scientific knowledge for social benefit is essentially what ABA is all about. ABA practitioners focus on changing behaviour. “Behaviour” is not shorthand for problematic behaviours. Although ABA practitioners do work with people to help reduce problematic behaviours, much of the focus of ABA when working with children with autism is on building new behaviours, new skills, and so helping children to learn new things.

One of my colleagues is fond of saying that whenever we achieve some change in behaviour, the principles of learning will have been brought into play. The laws of learning are universal so whether we intend to use them or not when people learn, when behaviour changes, this happens via underlying learning processes.

Behavioural interventions, drawing on those principles of learning and behaviour change, also feature in many contexts to help improve quality of life. For example, in the UK, the National Institute for Health and Clinical Excellence (NICE) has recommended a number of behavioural interventions in different areas of health and well-being. These recommendations include:

• Dialectical Behaviour Therapy for Borderline personality disorder (Jan 2009)
• Behavioural Activation for Depression in adults (Oct 2009)
• Multi-component intervention including behaviour change, and goal setting for Obesity (Dec 2006)
• Behavioural parent training in Parent training/education for child conduct disorder (July 2006)
• “Behavioural and functional analysis” for challenging behaviours in Dementia (Nov 2006)
• Behavioural intervention to improve daily living skills and for challenging behaviours in adults with autism (April 2012)

So, ABA is all around us, and is used in a variety of contexts. In addition to ABA interventions in health, social care, and education, ABA practitioners have used their skills to (amongst other things): improve safety behaviours, train elite athletes, and to increase productivity in work settings.

It has often been said that neither of the As in ABA are for Autism. Hopefully, it is clear by now that ABA is not an intervention for autism, but can be applied to help children with autism.

Defining ABA for autism

The following single sentence might be a good way to capture the essence of ABA when applied to working with children with autism:

“ABA is a values-driven, child-centred, developmentally-informed, evidence-based, effective use of principles of learning to help children with autism achieve their full potential.”

The core values and commitments of ABA in the field of autism are:

• ambition for the child, and optimism about what is possible for them
• an assumption of every child’s ability to learn (not dis-ability)
• empowering the child by establishing skills and supporting management of their own behaviour
• collaborative working with other professionals, family members, and carers to best support the child

A further characteristic that is important to emphasise is that ABA does not “decide” on what behaviours to change or what skills should be developed for a child with autism. ABA practitioners are skilled at teaching skills and decreasing problematic behaviours, but the behaviours to be changed need to be externally defined by others. When working with children with autism, there are perhaps three sources used to identify the foci for intervention:

1. Children themselves, their families, and others around the child identify what outcomes are important to them
2. An understanding of typical development (what key developmental skills has the child developed, or what they need to learn next), and also the requirements of statutory educational curricula
3. A focus on pivotal behaviours that will facilitate further development: teaching communication, social skills, daily living or academic skills that can support independence and choice-making

Comprehensive ABA intervention for children with autism

There are many ABA-based short-term focused interventions carried out when working with children with autism. A particular skill might be taught over a few days or weeks, or a particular behaviour problem might be the target for reduction again over a few weeks. Behavioural interventions are used probably in the majority of educational settings for children with autism in this focused way.

The main debate in the field of autism about ABA is actually focused on comprehensive ABA educational models. These comprehensive models bring together multiple elements of ABA intervention in a co-ordinated way and share probably all of the following features:

• Teaching multiple, rather than only one, skill areas
• Breaking down complex skills into small, teachable steps to facilitate learning
• Teaching over a long period of time – years, rather than weeks or months
• Can be used for all ages of children and adolescents
• The intervention model includes structured learning opportunities and also incorporates “natural” opportunities across much of the day/week
• Involves a team of people (often including parents)
• Can take place at home or in school and other settings

What may be a surprise to some is that I have not included a prescription of intensity of intervention in this list of shared features – no mention of the number of hours per week that must be delivered. My perspective on this is that a focus on the number of hours of intervention is linked to when comprehensive interventions are used as an early intervention model. When used as an early intervention model, comprehensive ABA intervention is often called Early Intensive Behavioural Intervention (or EIBI). When intervening early, accepted wisdom is that one should deliver as much intervention as possible. In fact, this is slightly more than accepted wisdom. International expert consensus in the field of autism is that intervention should ideally be early and the more that can be delivered probably the better in terms of longer term outcomes. So, autism experts (not ABA experts alone) recommend early and reasonably intensive intervention.

Again, I see no prescription of intensity for a comprehensive ABA intervention. However, following international consensus, one would want to deliver as much high quality intervention as possible when delivering EIBI. If the resources available allow the delivery of 30-40 hours of intervention per week, one would want to do this. If the resources allow for only 5 hours of intervention per week, this should (in my view) be done too at a quality level as high as possible to ensure the biggest impact for that input.

ABA and autism after the early years

EIBI models have been developed to be used early in the lives of children with autism. The focus is on behaviours and learning that are developmentally appropriate for younger children. The teaching methods used are suitable for young children. The outcomes of interest are skills that might enable the child to benefit as much as possible from the rest of their educational experience in school, ideally within an inclusive educational setting. In fact, these are probably the general goals of all early intervention models for children with autism not just ABA.

Early intervention should, in my view, be early. However, comprehensive ABA educational models are also relevant for older children and adolescents. Although these models for older children will still share the features of comprehensive ABA interventions listed earlier, the content should look different. For example, the skills being taught to adolescents should be relevant to the adolescent stage of life and the preparation for leaving school. This is likely to involve foci such as money skills, independent use of public transport, accessing leisure facilities, and preparation for work.

Autism evidence 2. Using an evidence tool to draw your own conclusions about autism interventions

 

Please read the first Blog in this series [Who do you believe? A model for understanding the development of evidence for interventions in autism] before reading this blog.

In practice, Thornicroft et al. (see model above) point out that there are three points along the evidence continuum that they call blocks – where evidence earlier in the storyline tends to get stuck. Many basic science findings are not translated into interventions at all. This has often been pointed out in the field of autism – there is a great deal of basic and experimental research on children with autism, and much less use of this research to inform new interventions. A second translational block is when research demonstrates that an intervention CAN work, but then everyone relaxes thinking that is enough to now get on and inform service delivery. Finally, even when interventions are also tested in effectiveness studies research on scaling up and delivering interventions within typical practice and service settings is often just not done.

Using an evidence tool

By this point, I hope that you can see how you could use Thornicroft et al.’s evidence tool. What it will hopefully prompt you to do is to consider what question a particular research study is asking about an intervention. The Phases of the model help to clarify the questions.

So, is someone describing to you an early study designed to generate initial data on a recently defined new intervention (Phase 1)? If so, the research design might not be very strong or sophisticated. However, the research is still likely to be important. You should be asking about the theory underlying the intervention. The researchers or practitioners should be able to articulate this clearly and point to the evidence that supports their theory. A “good idea” is simply not enough – it has to be more properly considered.

If you are being shown results from a Randomized Controlled Trial, at what phase is this study along the continuum? Is it an initial pilot trial (in which case it is not yet definitive evidence), is it testing out whether an intervention CAN work, is it testing effectiveness in more typical settings, or is it even using a RCT design to test aspects of delivery (Phase 4) for an intervention with already established efficacy and effectiveness?

In practice, you will find that the evidence for interventions does not always follow the full continuum. Sometimes, this will be because those peddling an intervention have not fully understood what “evidence” really means. Sometimes, it suits those whose livelihood relies partly on you being interested in a particular intervention to give you only a part of the story. People may not be misleading you explicitly – they are simply not presenting the whole story, or do not have a way to easily communicate that full story to you. Now YOU have a way of looking at evidence yourselves and can start to draw your own conclusions.

There are also good pragmatic reasons why the development of evidence may not follow through the full set of phases. First, this could take a very long time in total. It is often been pointed out that the journey from basic science to practice can take two decades or more. Thus, it is perfectly appropriate to take pragmatic decisions about what evidence is most needed now. Obviously, we cannot sit around waiting for 20 years to develop evidence-based interventions for children with autism. We need them now (or soon). This is not an excuse for poorly designed research. However, researchers and practitioners could be more transparent about what they are doing and why and could use the evidence tool to explain the bigger picture and, therefore, the limitations of the evidence that they currently have.

Some examples

At this point it might be helpful to show how I might use the evidence tool to talk about the nature of evidence emerging from individual intervention studies in the field of autism. I have chosen two examples that use widely accepted strong research designs (both use RCTs), but their focus is quite different.

Howlin and colleagues (2007) carried out an evaluation of PECS (the Picture Exchange Communication System). 18 classes of children with autism were allocated randomly to PECS or education as usual. Teachers and parents received a 2 day PECS training workshop, and PECS trainers then made 6 half-day consultation visits to each class over a 5 month period. Although communication initiations and use of PECS increased in the PECS intervention classes, there were no measured increases in scores on standardized language assessments for the PECS group compared to the control group. The increases in communication initiations and use of PECS also did not continue after the consultation support ended.

I think the Howlin study of PECS is firmly in Phase 3 of the evidence continuum. This is a study of the effectiveness of PECS under typical conditions, delivered by school staff after minimal training, with some supervision, in typical school settings for children with autism. Questions that we might want to check up on, looking at this study on its own, might be about the theory underlying PECS, whether there are efficacy trials already published, and how could the intervention be scaled up and included as a standard part of practice for children with autism in educational settings (whilst also delivering outcomes)?

What is clear is that a study like Howlin’s is simply a part of a story and is not the whole story.

Another recent and well-publicized study is from Dawson, Rogers and colleagues (2010) evaluating the Early Start Denver Model (ESDM). In their research, 48 young children with autism were allocated randomly to EDSM or to “treatment as usual” for young children with autism in their locality and received intervention over 2 years. The intervention was delivered by highly trained and experienced graduate staff supervised by experts. The research team also included the developers of the intervention approach. The results were positive. For example, cognitive skills and adaptive skills were improved in the ESDM group compared to the control group.

The Dawson and Rogers evaluation of EDSM for me is a Phase 2 study – it is designed to test whether ESDM can work under ideal and fairly tightly controlled circumstances. It was an efficacy trial. The intervention is also based on basic developmental research in autism, is manualized, and has been tested previously in other smaller scale studies (see Phases 0 and 1). However, it would be a mistake to suddenly suggest that everyone must start delivering ESDM as the model for pre-school services for children with autism. Effectiveness studies are needed and, more significantly, research is needed on how ESDM could be implemented as a part of a broader pre-school educational service and on a large scale.

Again, the Dawson and Rogers study is very important and a great piece of research, but it is only a part of the story.

A call to researchers and expert practitioners

I would encourage researchers and expert practitioners to use the evidence tool to summarize the evidence for particular educational and psychological interventions for children with autism. To properly critique any intervention, evaluate research evidence from basic research findings right through to any effectiveness and implementation studies. So, when you are explaining the evidence base for an intervention, see if the evidence continuum tool helps you to provide the full story.

In a later blog, I will provide a perspective on the evidence for Applied Behaviour Analysis intervention using the evidence continuum tool.

In addition, when reporting an individual study, researchers may find it helpful to refer to the evidence continuum tool to place their research question and their findings in context.

Autism Evidence 1. Who do you believe? A model for understanding the development of evidence for interventions in autism

Many experts regularly pontificate about which educational and psychological interventions are “evidence-based” for children with autism. The problem is that there is no one right answer. So, there can be as many perspectives on the evidence base in autism as there are experts. Who do you believe? The purpose of this series of blogs is to describe a tool that may help you draw your own conclusions and/or to empower you to ask those experts more probing questions about the evidence.

A continuum of evidence

All psycho-educational interventions for children with autism are complex – they normally involve several components, might target several outcomes, involve the child and other people, and will typically occur in several environments and over a period of time. Several people have pointed out that it can be helpful to think about a process for the development of evidence for complex intervention that can be represented as a continuum. Thornicroft, Lempp and Tansella (2011) described a model of the process of development of evidence that I find very helpful 

[see http://journals.cambridge.org/action/displayAbstract?fromPage=online&aid=8361312&fulltextType=ED&fileId=S0033291711000109]

I have re-drawn Thornicroft et al.’s mode above representing their phases of evidence development and emphasizing increasing evidence from left to right.

The idea of this model is that basic science or theory should inform interventions in the first place. In essence, this is because any intervention should be informed by the latest understanding of a “disorder” or difficulty and interventionists should be able to say what they are trying to do! The fact that Phase 0 is called Basic Science is important. For me, this means that it is not enough to be able to describe what you are trying to do with an intervention – your story needs to be evidence based. For example, you might be informed by common difficulties or strengths in children with autism, the way that children with autism learn, or typical child development (i.e., what are the key early skills children with autism need to learn to enable them to build more complex skills).

An intervention should also be very clearly defined or described (Phase 1) and ideally written down in detail in the form of a manual for those using the intervention. The competencies needed to deliver an intervention probably also need to be described and training/supervision models developed. There would also need to be ways to check whether someone is properly delivering an intervention (this is called “fidelity”). A new intervention can then be subjected to initial testing – trying it out, and measuring some outcomes. What outcomes to measure would also be closely related to the underlying theory – what are you trying to change?

At some point during Phase 1 and in preparation for Phase 2, we might also expect to see a Pilot Randomized Controlled Trial (RCT) conducted. Using this research design, children with autism would be randomly assigned to receive the intervention or to receive another intervention (e.g., existing practice) or even no particular intervention at all. A pilot RCT tests out everything ready to run a definitive RCT at Phase 2. Pilot RCTs are not designed to answer the question of whether an intervention works.

During Phase 2 of the development of evidence, research studies (probably large scale RCT studies) are carried out to ask whether the intervention can work. This is a question of efficacy – given well-resourced, well-trained and supervised interventionists delivering an intervention with fidelity, CAN the intervention be shown to work? It is very important at this stage to recognize that studies are not designed to tell you whether an intervention will work when rolled out into practice.

Too often, the development of evidence in the field of autism stops at Phase 2. Or at least, experts focus on this aspect of evidence when they define what are evidence-based interventions. From Thornicroft et al.’s model, you can see that this is far from the end of the story. The next step at Phase 3 is to carry out tests of the effectiveness of an intervention. CAN an intervention work is an important question, but whether it still works in practice and under less ideal conditions is the question of whether the intervention can be effective. Phase 3 research studies will often be RCTs again. The focus, for example, might be on whether if you train people who train others to deliver the intervention and they do this in typical school settings using teachers or classroom assistants, do you still get positive outcomes.

Phase 4 for me is about then how do you scale up the delivery model to deliver the intervention as a part of standard services. In health contexts, people often talk about care pathways. So, how can the new intervention be incorporated into an existing care pathway and still be delivered in a way that leads to positive outcomes? What conditions are needed in practice to make sure that the intervention still “works”? In educational contexts, questions such as how an intervention can be delivered in classrooms and how does it dovetail with the curriculum? All of these sorts of questions need to be subjected to research study.