A case study in demonstrating a return on investment
School budgets are under increasing pressure so it has never been more important to demonstrate a return on investment when it comes to purchasing new technology, new software or introducing a new initiative. But all too often providing an accurate, quantitative impact evaluation proves mighty difficult within a school setting where teaching and learning tools are rarely used in isolation, which means that impact assessment is often based on gut feeling.
Gut feeling and anecdotal feedback can often be the best form of evaluation but aren't always sufficient to justify additional spending or renewal subscriptions.
Dominic Salles, Assistant Head at Chipping Campden School, a state secondary school in Gloucestershire wanted to find a more objective way of evaluating the impact and determining the school’s ROI on a recent subscription to an online learning tool.
He explains: “We introduced GCSEPod in to school in December 2012 because we were interested in the concept of embracing mobile learning. The GCSEPod service provides audio-visual learning and revision content designed for use on mobile devices. Students across the board were downloading the podcasts and anecdotal feedback from students and staff indicated that it was a useful resource.
“Our exam results that year, 2012 - 2013 increased. Some might say…. well, there’s your answer, GCSEPod must be working. Whilst my gut feeling told me that this was the case, I knew that the improvements in our exam results may well be in part due to GCSEPod, but that success was due to a combination of efforts and resources, including the dedication of our teaching staff and students alike.
“The time was coming to renew our subscription to GCSEPod and whilst there is growing support for it across the school, I knew to gain the full backing of the senior team I should try to demonstrate the positive impact it has had on the students’ learning as well as directly on their exam results.
“As a school we use Fischer Family Trust (FFT) data in order to raise standards and aspirations even further and whilst I appreciate that this data is loved by some and loathed by others, it does provide useful insightful information on which to set benchmarks and raise aspirations.
An emerging pattern
“I decided to take this data and look at how individuals performed against their FFT score and compare this to their use of GCSEPod. In essence I was looking to see if a pattern might emerge; would students who downloaded a greater number of podcasts be more likely to outperform their FFT scores or would this have little bearing?
“I looked at individuals across four core subjects where GCSEPod had been well utilised- German, French, English and Science.
“I looked at the amount of downloads versus a student’s VA against target, the VA against average of all grades, the percentage VA above FFTD target and finally percentage VA against average of all grades.
“In every subject a pattern emerged. The students who had downloaded only a handful of podcasts had slightly underperformed against all four criteria whereas the students who had downloaded the most podcasts had greatly outperformed against their FFTDs, often achieving a full grade higher than expected.
“This armed me with quite powerful evidence that our subscription was delivering a return. But then what’s to say that those students who downloaded the most podcasts are not simply studious and conscientious natural achievers and would work hard to achieve their best regardless of the tools available?
“To ensure this wasn't the case, I looked at the results that these students achieved in subjects where they did not have access to GCSEPod and used alternative revision techniques. In every case, the students downloading in excess of 50 podcasts in any one subject, outperformed their FFT score more significantly than they did in subjects where they hadn't used GCSEPod.
“The data collected from carrying out this evaluation provides the evidence that we need as a school to know beyond reasonable doubt that our recent investment is delivering a return and helping our students to achieve their potential.”
This assessment using FFTD or similar resources could be used to evaluate the impact of a whole host of initiatives across school. Dominic Salles would be happy to share his evaluation tool with other teaching professionals looking to determine the effectiveness of tools and resources they have introduced in to their schools.
For further information contact www.gcsepod.com