My friend Niamh Ryan, joins me to talk about evaluating teacher training. How can you tell if your training is making a difference?
Welcome back to the TEFL Training Institute podcast everyone, I'm Ross Thorburn and this week we're talking about evaluating training. My guest Niamh Ryan and I are going to talk about, well firstly, why it's important to evaluate training, the different levels we can evaluate training at and how by going through the process of evaluating training we can actually end up making our training much more effective. Enjoy the episode.
Welcome back to the podcast Niamh. Why is it useful for trainers to think about evaluating their training? Well if you're doing training there's a cost. There's a cost to your time, there's a cost to the people's time, there's a cost to the organization like from putting it on and if you're investing evaluating training is going to justify that that investment is worthwhile and that is going to be from a learner's perspective in that that they are developing learning new skills or it could be from a business perspective in that yeah like actually it's solving the problem that we're trying to solve.
Yeah I completely agree. I think there's an aspect of job security about this that if you can show your school or your company that your training works and was worth the cost then you've helped to make your job more safe. But for me there's also another really important part of this which is that by evaluating your training you can also find out what it is that you need to do to make your training better.
So finding things that either stop your trainees from learning or stop them from applying what they learned is going to help you improve as a trainer. Yeah exactly yeah training evaluation is difficult because it's almost never the only intervention that the the student got. So if a bunch of teachers go on a teacher training course is it all to do with the training is it the fact that they did this great training but they're also in a really great school that actually gives lots of feedback so there's lots of extra things at play so it can be hard to isolate the impact of training but just because something is hard doesn't mean we shouldn't do it.
Okay so tell us about what sort of model do trainers usually use for evaluating training? Yeah so the seminal work on this is by Donald Kirkpatrick and that was done in the 1950s but still very relevant today and he talks about four levels of evaluation. So the first level is reaction level and that's just you know did the learners like the training you know how do they feel about the training. Level two then is learning itself so that would be really looking at achievement of the learning objectives so if you set out to do x y and z can the the learners actually do that and generally those are easier ones to do because the happy treat is easy to administer and the learning objectives with a bit of thought you can do a test or you you can do an exercise that actually tests those learning objectives and you should have evidence that those have been met.
With past that then we go to behavior and outcomes level and our business impact level and the behavior piece is really thinking about transferring so after a training program are people transferring what they learned in class back to the workplace or the situation that they're in and even you can think about that with students you know if they learn a particular structure in class like do they actually use that in their conversations when they're in a free conversation and it's really challenging to get people to apply what they've learned and so thinking about how would you see that you know a typical way we think about teacher training would be to ask the teacher's manager did you see a difference when they went back and then that fourth level is really thinking about outcomes and does that have an impact on the business and so an example might be something like diversity right so say we said okay we want to do a program to develop leaders from diverse backgrounds and you would put this cohort through and you would do the first three levels so you know did they like it did they learn from it did they apply it but the business outcome might be okay do we actually have more diverse leaders on our senior leadership team than we did before right so that's a business outcome and then even further than that now that we have that diverse leadership team does that actually benefit our business is our business performing better because of that so it's really kind of tapping into the business and I think as learning professionals we really do need to partner with the business to understand what are those strategic challenges that they're facing like what are the problems that they're trying to solve and we work backwards from there rather than kind of coming up with an idea like oh I would love to run a training session on this good but actually if you're trying to really demonstrate an impact you're really aligning what you're doing with where the business is going and by understanding what they're trying to achieve that will help you to work backwards so you can measure the impact right so we really need to start off with a challenge or an issue and work back from there and a personal example I've had is for example a school has come to me before and said our problem is that parents of students aren't renewing their contracts and they say it's because their kids have studied here for a year or two and they've not learned enough so in that situation the aim or the outcome that you want is parents staying longer but let's say after observing a few classes I found that students weren't really being challenged very much in class so then the next step is to work back from that and design training which trainees enjoy and they can learn some strategies or techniques to challenge the students more and then after that you need to check that they've actually applied those to their teaching and then finally if all those things go into place you check well does the student retention actually increase so let's dip into the four levels in a bit more detail the first level is finding out trainees reactions to training why is it useful to measure if people enjoyed the training or not so I think there's two bits you want to separate out the reaction level and then the learning levels and actually just to kind of point out the criticism of Kirkpatrick's model is that actually these levels are not correlated right so you could really like a training but learn not very much from it right because you could be having a great day with your pals out of work on a training course but having said that even other models that I've seen they do tend to kind of look at these elements of it so really thinking about how engaged the trainers were could they transfer it and then looking at the impact of it so in terms of reactions to the training that's just going to be a little bit of what worked well what didn't work what did you like did you like the training did you like sandwiches did you like the room etc right and you know if you want to get your 100 compliance you'll give it as a paper one just before people walk out downside of that of course is you want to please the trainer or maybe you don't want to write down anything negative about the trainer and so if it's not anonymous and you're handing something back to the trainer you mightn't actually get a a very honest outcome but look I think you can do that in a kind of a quick way like on a scale of one to ten how happy are you with the lesson today you could hold up a number these levels are very much for the trainer right to see how things are going what's going well what's not going well what should I change tell us more about the second level how can we measure what people learned so in terms of learning what someone says they learned and maybe what they actually learned might be a little bit different like so for example if you're a trainer and you have just covered the last thing in your your class likely the person is going to say I learned that because that's the last thing that they remember right our brains tend to remember what we did first and what we did last just because I remember it does that mean that I've actually learned it or understood it so as a trainer you're trying to see can the person do these things from the learning objectives you need to create a situation where you observe them doing it rather than they report that they do it so a test is a way to do that and you know that works well if it's a knowledge thing you know but actually for more complex learning true or false or you know multiple choices tests probably not going to get into the depth of the understanding so it might be a case of get them to do an assignment and actually if you look at a lot of the way college courses are set up nowadays is less focused on exams and more focused on these assignments because the assignments will allow the student to bring in what they've learned and then you can see that they can apply the concepts so I suppose it's just thinking a little bit about what could you get your learners to do that will show you that they have mastered this right so that could be getting trainee teachers to plan a lesson and apply some of the ideas from the training in that lesson as the trainer you can then see if they've been able to apply what they learned at least in the training room and I suppose the thing that's interesting here is that evaluating training is really something that's part of the training rather than separate to it doesn't need to be something that you do afterwards this is really part of the training process in itself absolutely and and that that's a really good way to think about it because actually by doing these evaluations you're actually consolidating the learning right and you've more chance then of hitting the level three of helping them transfer because they're revising things they're practicing things and you're trying to create a situation that's as close as possible to what they would actually have to do when they go into a class so the next level in this Kirkpatrick model is whether trainees can actually transfer what they've learned and use it in their daily work does it become a behavior I know you said earlier that one way of checking what trainees learned in training is by checking in with their manager one of the issues that I find with doing that is that language school managers well for a start they don't get to see everything that the teachers do so suppose something else you could do there is ask the trainees themselves after training tell us a bit more about how trainers can do that yeah you know you might check in with them three months later over the last three months where did you have an opportunity to apply this what went well what didn't go well so really helping the person to reflect on the situation right so they can really think about the transfer or you might set some homework so like an example might be if you have teacher managers coming to a training session and you want to teach them about giving feedback you've talked about feedback in your training session you've done some activities about it and then your homework is okay I want you to give a colleague or a team member some feedback and I want you to reflect on that situation so you know in that situation it's great because it's unlikely that they wouldn't if they're a teacher manager have an opportunity to apply that but it's really kind of being deliberate about that opportunity and if your team members are reporting that you're actually doing these things like that's much more valuable information than you saying that you're doing these things right because actually when you're learning to be a manager you're learning how to be better for these people and if they're telling you that you're better now then that's really good feedback I think what's fascinating about this is that by going through that process of seeing what your trainees were able to use you're then giving your trainees an opportunity to tell you about what they were able to do and what they found challenging and that then allows you the trainer to coach them or help them through that something I often ask teachers to do is to take a video of them using something from the training in one of their regular classes and share that video with me and the other teachers who were in the training this is an online group somewhere and that also lets the other trainees see how someone else applied what they learned so just by going through this process of evaluating the training you actually I think can make the training a lot more effective now tell us about the last level Niamh the idea of evaluating the outcomes of training how can trainers go about that when we're thinking about business impact like if I'm a CEO of a company or a managing director and I'm making money available for training I'm probably not that interested in the minutiae of the learning objective but I am interested in the program as a whole then you start to think about how am I going to check that this is working can I have a group that's going through in a group that's not like so if you think about schools it might be teachers in school a got it and teachers in school b didn't get it and you can look at the impact there or is there a way to just administer a quick test at the beginning and then three months later can you do that as well that will also help them with the business outcomes because what are the outcomes that are coming from that and this comes from understanding what is the problem that you're trying to solve why have you spent the time developing this training program in the first place now I think a lot of what we've been talking about is really for evaluating an entire training course but I would guess for a lot of the people listening they might not be running an entire course they might just be running a workshop once a week for their teachers how can trainers who are doing that evaluate how effective their training is so I suppose it comes back to why are you doing that if I was asking you I mean you you've obviously had a role where you've done that before why did why did you do CPD with teachers well I would say first it's probably because the teachers just expect some professional development and we're probably trying to make the teachers feel like they're learning and developing in the job because most people want to get more out of their job than just money well if you're telling me that my mind is immediately thinking about them and that's related to teacher engagement it might be part of your employer brand so it might be a reason that teachers choose to come and work here versus somewhere else because there's good development you're not going to stop it so you can test oh what happens when I take this away there's enough research out there to know that actually providing these programs to teachers is important but what we probably would be looking at is what I'm doing is that effective and what we would really want to understand from that is okay when we say it's teachers expect that what do they expect are they expecting skills development so that they become excellent teachers or are they expecting CPD that helps them to be promoted or to progress so maybe your evaluation there is really return on expectations do we think that teachers are staying longer because they're getting this development are we seeing in our schools are we seeing teachers progressing faster than in other schools because they get this development are we seeing that our student satisfaction scores are better because our teachers are are better so there's still an opportunity to step out of you know I'm doing CPD because teachers expect and then using those same teachers to say how do we make this program even better like then you start to create something that's can be really motivating for the people going through it one more time everyone that was eve ryan for more podcasts from us check out our website www.tefltraininginstitute.com thanks again for listening we'll see you next week goodbye