A continuum of evidence
All psycho-educational interventions for children with autism are complex – they normally involve several components, might target several outcomes, involve the child and other people, and will typically occur in several environments and over a period of time. Several people have pointed out that it can be helpful to think about a process for the development of evidence for complex intervention that can be represented as a continuum. Thornicroft, Lempp and Tansella (2011) described a model of the process of development of evidence that I find very helpful
I have re-drawn Thornicroft et al.’s mode above representing their phases of evidence development and emphasizing increasing evidence from left to right.
An intervention should also be very clearly defined or described (Phase 1) and ideally written down in detail in the form of a manual for those using the intervention. The competencies needed to deliver an intervention probably also need to be described and training/supervision models developed. There would also need to be ways to check whether someone is properly delivering an intervention (this is called “fidelity”). A new intervention can then be subjected to initial testing – trying it out, and measuring some outcomes. What outcomes to measure would also be closely related to the underlying theory – what are you trying to change?
At some point during Phase 1 and in preparation for Phase 2, we might also expect to see a Pilot Randomized Controlled Trial (RCT) conducted. Using this research design, children with autism would be randomly assigned to receive the intervention or to receive another intervention (e.g., existing practice) or even no particular intervention at all. A pilot RCT tests out everything ready to run a definitive RCT at Phase 2. Pilot RCTs are not designed to answer the question of whether an intervention works.
During Phase 2 of the development of evidence, research studies (probably large scale RCT studies) are carried out to ask whether the intervention can work. This is a question of efficacy – given well-resourced, well-trained and supervised interventionists delivering an intervention with fidelity, CAN the intervention be shown to work? It is very important at this stage to recognize that studies are not designed to tell you whether an intervention will work when rolled out into practice.
Too often, the development of evidence in the field of autism stops at Phase 2. Or at least, experts focus on this aspect of evidence when they define what are evidence-based interventions. From Thornicroft et al.’s model, you can see that this is far from the end of the story. The next step at Phase 3 is to carry out tests of the effectiveness of an intervention. CAN an intervention work is an important question, but whether it still works in practice and under less ideal conditions is the question of whether the intervention can be effective. Phase 3 research studies will often be RCTs again. The focus, for example, might be on whether if you train people who train others to deliver the intervention and they do this in typical school settings using teachers or classroom assistants, do you still get positive outcomes.