Emma Dorn | McKinsey & Company - Transcript
Jeremy Singer: [00:00:00] I'm Jeremy Singer, president of the College Board, and this is the Education Equation. I've spent my career grappling with what truly drives student success. On this podcast, I'll talk with people who are researching, building and scaling solutions that matter. Every episode will go beyond the hype and focus on data and evidence to see what's actually working.
Let's stop guessing and let's figure out what works.
My guest today is Emma Dorn, a senior knowledge expert at McKinsey & Company and one of the leading voices on how school systems improve or fail to improve at scale. Many listeners will be familiar with Emma's work from McKinsey's research on pandemic learning loss and unfinished learning, which helped shape how many districts thought about recovery.
She's also co-author [00:01:00] of "Spark and Sustain," a global study of how school systems make progress and, just as importantly, how they then sustain that progress. What makes Emma's perspective especially valuable is her international lens. She looks beyond the US education debate to systems operating under very different constraints, including many in the Global South.
And she asks a practical question, not just whether an idea is promising, but whether it can actually work in the broader world at scale for students and teachers. Emma Dorn, welcome to the Education Equation.
Emma Dorn: It's a pleasure to be here.
Jeremy Singer: So today, we're gonna do something a bit different. we're gonna walk through a number of lessons or case studies from global examples where technology seems to be solving a real instructional or implementation problem.
For each one, we're gonna discuss what's the lesson, the takeaway for our listeners, what's the evidence that supports its efficacy, and finally, most importantly or equally importantly is, are there things that our listeners can do to apply [00:02:00] those lessons, at their school, at their district, in their classroom?
But before we jump into that, I wanna talk about why we should look beyond the US. And I hope this isn't prevalent among listeners here, but we do know there are a lot of people skeptical about what can we learn outside of the US, or what should we learn outside the US. From your global work, what do we miss if we focus just narrowly on US debates and US examples?
Emma Dorn: Yeah. So I think there's a lot to learn from outside the US, but I would add the caveat, we do need to be really careful with lifting and shifting from one school system to another. That rarely works. However, I think there's some learnings where we can think about how systems change from poor to fair to good to great, and how the interventions that you need to put into place change across that journey.
And so as we're looking right now from the Global South, I think we need to be a little bit careful because they're at a very different stage in their journey in educational improvement. That said, I think there's a few things that, that I've seen coming in the Global [00:03:00] South that's really exciting and interesting.
One is that there's a lot more disruptive innovation going on in low-income countries. This may be, partly because some of these areas are terribly underserved right now by incumbents, across low and middle-income countries. Right now, the World Bank is estimating seven in 10 children are in learning poverty.
That means they're unable- Wow ... to read a simple text by the time- Wow ... they leave elementary school. Yeah. In sub-Saharan Africa, that number is nine out of 10. And so what we're seeing in these areas that are underserved is that there- Moving towards more disruptive innovation instead of in the US we're seeing more sustaining innovation because the system is better served.
And so I think that then by looking outside of our borders, we can see some of these innovations and maybe then think about how we might bring them back.
Jeremy Singer: Yeah. That's great. I have a qual-- follow-up question, but very quickly just defining Global South for our listeners.
Emma Dorn: Yeah. So roughly low and middle income countries.
So that's Africa, India, Latin America, some of the lower middle income countries across Asia.
Jeremy Singer: Got it. [00:04:00] Great. So southern hemisphere, lower income countries. It's really interesting 'cause you're, part of the argument which I buy is some of these, the challenges may be even more stark in some international areas.
So re-really could provide some different insights. I've also heard you talk about how lower resource systems, which some of these countries by definition have, less resource, face sharper constraints, and then they have to therefore design, either be clear on their priorities or be more disciplined in the design to offset that.
Can you give a little more texture on that and, why that may be relevant and important way to learn?
Emma Dorn: Yeah, I think that's very true. And I think that in an environment with a lot of constraints, a lot of these systems do a couple of things differently. One is they're really designing for the core instructional bottleneck, not the technology.
So they can't afford just to kind of apply one-to-one computing for everyone. Right. They have to really think about what is the core instructional bottleneck we're facing and how can technology help that. And I think that in the Global [00:05:00] North, often the conversation is, "Do we use tech? Do we not use tech?"
And I think in the Global South, the conversation is much more targeted about what's the bottleneck and how can technology help to de-bottleneck. So that I think is one thing we can definitely learn from. I think the other thing is just when you're working under constraints, it can make you more innovative, and it also makes you more focused on the evidence base.
So you see in the Global South a lot more RCTs, a lot more focus on what do we know from the evidence before we then try to a-apply that at scale. And so a couple of examples of that, that I think we'll go into here is focusing on technology for teachers, not just direct to student tech, because a lot of the evidence base that we've seen globally has shown actually greater impacts on student outcomes when you focus on teacher tech.
Jeremy Singer: Great. And we'll get into some of those lessons. I don't wanna do too much, but there's a David and Goliath quality to this. Yes ... David had to be more inventive than Goliath was for-forced to be. So I'm also thinking of all these montages of sports movies where, you know, before the big game or [00:06:00] meet, they go back to the very basic workouts.
they forget all the technology. They go to the core gym and work out. So anyway, let's jump in, though. So okay, so we're gonna go through each lesson. I want you to start. I'm gonna, I'm gonna tee it up, and you're gonna talk about the lesson, the takeaway. Then I want to talk about the evidence, and then at the end of it, we'll come back and we'll say, "What are the applications potentially in the US?"
First lesson, number one, design for the instructional bottleneck, not the technology. You mentioned it a minute ago. So explain to listeners what does that mean.
Emma Dorn: Yeah. So I think if you think about the context in many of these low-income systems, they have large class sizes, 50, 150 kids per class. They have uneven teacher preparation.
Some of them have no electricity or limited electricity, very low device penetration, very limited connectivity. And so they really have to think about what is it that we're going to do with technology that can actually design for some of our core challenges, and the core challenge in many of these systems is foundational literacy and numeracy.
A great example of this is, Imagine [00:07:00] Worldwide. They have designed and they work with governments to deliver child-directed adaptive learning through a solar-powered tablet. And so that's a tablet that is fully solar powered. It doesn't rely on any electricity. It's fully offline. It doesn't rely on any connectivity, and it's loaded with One Billion's One Course software, which really focuses on foundational literacy and numeracy, which is the fundamental need.
And so it's designed to work in these low resource environments where there's, maybe one teacher and 150 kids, and the teacher is not possibly able to get around those kids. So students will use the software for about 30 to 45 minutes a day, and the tablets will rotate across several students in the course of the day.
And just like adaptive software here, it adapts to each child's level, pace, and progress. And what's really exciting about this is, I think two things. One is that there's real evidence of effectiveness, but two is that there's real evidence of scale. And so the tablets now, there've been, I think, nine randomly controlled trials [00:08:00] conducted on this software, which I think is very dissimilar to a lot of the work that we see in the US, where it's much more quasi experimental or more kind of qualitative Evidence, real RCTs.
In Malawi, for example, they've been rolling this out. They're reaching nearly a million students at this point, and the percentage of learners attaining emergent or fluent status in reading doubled, and in mathematics doubled as well. So more than doubled in reading, doubled in mathematics. And so you're seeing at scale evidence of real improvements in student outcomes.
Jeremy Singer: That's amazing. And, for listeners, I, think we've talked about it before, but a randomized control trial is, sort of the best, it's just sort of the gold standard in some of this research. And so it is very hard in education to do an RCT. So that there's been nine is, like on the evidence scale, that's pretty convincing.
In this example, in Malawi specifically, going back to the lesson about you need to address the bottleneck, is it bottleneck or is it all the above? Is it lack of teachers? Is [00:09:00] it inconsistent instruction? Is it lack of student practice? Is it, lack of electricity? What would you say the bottleneck was that this designs for?
Emma Dorn: I mean, I think the fundamental bottleneck that this designs for is just the fact that the teachers cannot possibly meet the diverse needs of 150 kids. And one of the things we've seen across Africa is access has improved tremendously over the past couple of decades, which is fantastic. But as access has improved, class sizes have ballooned, and it's just really difficult to teach that many kids.
And so that's the primary bottleneck, and then all the other bottlenecks are just a whole bunch of constraints around then designing technology that can support that, that can overcome that.
Jeremy Singer: Yeah. Yep, And so if I'm a, instructional leader at a district in the US, from this example, what would I take away?
Is there a specific-- like when you think of bottlenecks that are facing the average, to the extent there's an average US, school or district.
Emma Dorn: Yeah, I mean, I think it's a little different in an average US school or district because class sizes are so much smaller. And we know [00:10:00] that, maybe between 25 and 30 kids, it's not gonna make so much of a difference.
Between 30 and 150, it's just a- Right. Right ... ball game. But I do think that there's similar adaptive learning technologies that can be really transformative in the US. And I think that the real difference there is that you have a lot more potential for the teacher to actually be engaged, to see the back end, to be able to be adapting their, day-to-day learning as well, which I think is harder to do in Malawi.
I think the other thing maybe that the US can learn from this, though, is fidelity to the model is everything. I think the reason that this has w-worked in Malawi is this is being rolled out as a government-led national expansion. It's gonna reach eventually over 3.8 million children across all the primary schools in Malawi.
Wow. And so there is real fidelity to this 30 to 45 minutes a day. And so I think what we see in the US is a lot more kind of incoherence, and one teacher's using one program, one teacher's using another program. Maybe they're not [00:11:00] actually aligned to the national curriculum. Maybe they're not aligned to standards.
And so to get The most out of a technology. For a technology to work, you have to use it
Jeremy Singer: Yeah, and there's a higher level of compliance or agreement or buy-in potentially in, in these examples, which it can be hard to replicate. So maybe a lesson is, can you rally the district, the teachers, the, aides, the, instructional leaders around something, to say, "Hey, we're gonna do this, and we're gonna do this with high fidelity"?
One last question before we jump to lesson two. Do you think this intervention, you mention it for really, like early literacy, early math skills, which we know are so foundational, so, so important. Do you think this kind of interactive set of tools are better for that than for more advanced skills, or is there any research to support one way or the other?
Emma Dorn: So the research doesn't really exist on the similar tool than to Imagine. They don't have, at the moment, a higher level tool for higher level- Yeah ... skills. I think it can be for both. I think one of the [00:12:00] things we forget is actually many kids in high school don't have these foundational skills, and so one of the things they're looking at right now is taking their model and using it for remediation for high school kids, which I think could be really effective as well.
Jeremy Singer: Yeah, and that's fair. The- they saw some stats on the number of, students, in high school that still need remedial math and reading is, quite high Great. So that takes us to the second lesson. A lot of AI and edtech conversations often focus on tools serving students, and, I know in my podcast I've interviewed a number of, people that are building tools for students.
But your second lesson is technology that's designed for teachers may have the opportunity to be even more transformative. So can you share why that is and, again, examples of that?
Emma Dorn: Yeah. I mean, I think that if you look at the historical data, it suggests that tech for teachers can have more impact on student outcomes.
We did a big analysis of the 2018 PISA assessment. This is the Program for International Student Assessment, which is done over about 80 countries, 500,000 students [00:13:00] every three to four years. And, what's really powerful about this assessment is it not only assesses kids across countries in math, reading, and science, but it also asks a whole bunch of questions of their teachers, of the students themselves, and of parents about resources, behaviors, and mindsets.
And what we found is, that tech in the hands of teachers added about 10 to 20 points to student outcomes in PISA, and that's like a couple of years, one to two years of learning. But tech in the hands of students actually reduced student outcomes by about 30 points. Wow. Now, this was 2018, and obviously we've learned a lot more about technology since then, and there may be re- many reasons for this, but one of them, I think, is the distraction that student-facing devices can cause.
In the 2022 PISA, 65% of students were distracted in their mathematics lessons by devices, and 45% said they felt anxious if their devices were not near them. And so for tech for teachers, we've seen this with AI as well, that often you can use these technologies more effective as an expert than you can as a [00:14:00] novice.
And so teachers are the experts, so scaffolding teachers to enable them to even better perform that kind of collaborative teaching experience for students, I think can be particularly transformative.
Jeremy Singer: Yeah, and it probably goes back to your point of high-fidelity implementation. it's hard to get teachers and school systems to implement with a consistency, but I imagine if there's any degree of freedom of the user to how to use it, you're not gonna get the same usage.
I wonder, I, just last week I think came out the first study in the US about- Removing w- where you can't use cell phones and what's the impact on learning. And I think I know I was disappointed with the results 'cause I was hoping we would see these spikes in improved student outcomes where devices were constrained and it's not bad, but didn't find any real growth in that.
And again, it's early and there's a lot more to do. But part of the argument would be the device was a distraction, you removed a distraction. Now, I've heard people posit, well, they still often have a, [00:15:00] computer that they're on, and so that may sort of cause a distraction. Any thoughts on, that study or anything else?
Emma Dorn: Yeah. I mean, again, it's this tech ver- versus no tech versus what is the tech actually for. If you can get locked down tech that is actually effectively doing adaptive literacy and math, that's great. If the kids are watching YouTube, not so much. And there are so many different technologies in our classrooms, sometimes it's hard to know exactly what each of them is being used for.
I, I think the other point beyond distraction that argues for tech for teachers is just that it's really cost-effective 'cause the resources just aren't always available to maintain one-to-one student devices, especially in emerging countries. And so if you can equip teachers with devices and equip teachers with the professional development to really use those well, I think that also is another argument for- Yeah
investing on that front.
Jeremy Singer: Yeah, that makes a ton of sense. And can you give a good example of a technology geared toward teachers and you've seen?
Emma Dorn: Yeah. So one that I think is really interesting is New Globe, and so their model centers on providing teachers with structured lesson [00:16:00] plans which really scaffold and improve their practice, but which also feed data systems to improve system performance.
And a bit of context is needed here, again, as you kind of think about from poor to fair to good to great. In poor to fair systems, these are the systems that, have very high learning poverty rates and more ill-prepared teachers, less human capital. Scripted lesson plans are one of the things that have been really meaningful in improving student outcomes.
It's an approach called structured pedagogy. It includes one-to-one textbooks, scripted lesson plans, teacher coaching. So we know that those work on paper. What New Globe has done is it has these teacher tablets optimized, again, for low infrastructure environments. Offline first, they sync to the cloud with 2G connectivity.
They have two weeks of battery life. And on these tablets, the teachers receive customized teacher guides for every grade, subject, and teaching day. And so these are aligned to the national curriculum. They use evidence-based instructional sequences. They have scaffolding, and they include, [00:17:00] not just the instructional tools, but not just the demonstration the teacher gives, but also questions designed to increase participation.
And so these have been really transformative in- A lot of African countries. I think for two reasons. One is because they're scaffolding the teachers to be able to deliver really good content, high-quality instructional materials. The second is, that they're part of this broader learning management and data collection system that actually captures is the teacher in the classroom?
Are the students in the classroom? Are the lessons being completed? What are the learning outcomes coming out of this? And this enables them to do some really interesting AB experiments to test instructional variations, so they can test shorter teacher guides, longer- Yeah ... teacher guides, and be able to link that to student outcomes to see what's actually working.
Jeremy Singer: Yeah, no, that's excellent. And so my sense is the, scripted lesson plans and, more directive help, as you said, I think raise the floor, and it's great for teachers with less experience or newer teachers. I know talking to teacher friends that say [00:18:00] it's also sort of it challenges, can feel restricting.
Is, that fair? Would you say that may work moving either a teacher with less experience or students that are way behind to, to, sort of move up on the spectrum versus really good at the top end?
Emma Dorn: Yeah. I think this is another example where you can't directly lift and shift. You don't want to be scripting a US teacher necessarily because the level of professional development that teacher's had, the level of pre-service training, the number of kids in the classroom, it's just dramatically different.
And so I think there are, however, ways that you can think of, well, how do teachers need to be scaffolded in the US? I think new teachers, maybe it would be great to have a little bit more materials that they can use and tailor, and this is where I think some of the-- and we'll talk about AI lesson planning in a little bit, but where you can base something off a historical number of lesson plans but then enable teachers to actually tailor that to their own classroom, I think is really exciting.
Jeremy Singer: Yep. Yep. Good, and we can get back to this in a minute. [00:19:00] Okay, let's jump to lesson three. Some of these can sound somewhat obvious, but are really important. So I think this one, technology works better when it's aligned with the curriculum, not sitting bes-beside it. So I think everyone would agree coherence matters, but give us more of why that is, is so critical.
Emma Dorn: Yeah. I think over the past decade, there's just been increasing recognition of the importance of coherence for aligning the assessment to the instruction to practice, aligning high-quality instructional materials to underlying grade level standards, aligning initial instruction to intervention supports, teacher professional development to all of the above.
I think in some ways it's self-evident why it's important, but the reality is it's just not happening in, in many places. I think there's data that suggests the average US district is using 1,400 different edtech tools every month. That can undermine coherence, and if we add AI onto that, if the models are content agnostic, they're just pulling from the entire internet rather than from very specific high-quality instructional materials, they could easily [00:20:00] generate lesson plans that look good on the surface, but that actually end up confusing students with conflicting approaches that don't follow the scope and sequence of the curriculum, or worse, pull from upon discredited pedagogies such as learning styles or 3-cuing.
And so I think there's, real potential then to say, as we're creating these AI-enabled tools, and the first two tools we talked about is more traditional edtech. From now on, these will involve some kind of AI component. The importance of coherence becomes even more important.
Jeremy Singer: Yep, yep. And you have examples where the good tools that have been aligned, examples of alignment?
Emma Dorn: Yeah, yeah. absolutely. So there's, an organization called Mentu, which is operating in Colombia and the Dominican Republic, and they have created an AI teacher assistant that supports lesson planning before class and reflection after class. And now this may not seem tremendously, leading.
We have a lot of different lesson planning AI tools out there right now, but what I think makes it special is a couple of things. One is that it's aligned both to the local curriculum and desired pedagogical goals, [00:21:00] so based, for example, on some of the World Bank teacher resources, and it both scaffolds lower-skill teachers, but it also enables them to actually have some agency.
And so it's based on all of the historical library of teacher lesson plans, et cetera, but it enables some tailoring, so teachers interact with structured forms in a chat-like prompt. And they don't just receive the lesson plan, but they receive strategy recommendations and ex- that explain why those strategies were selected so that it supports teacher professional development as well.
And what's been really interesting here, again, is the kind of scale of implementation. So it's being used system-wide in the Dominican Republic, aligned to their national curriculum, which I think, again, provides that consistency kind of across schools.
Jeremy Singer: Yep, yep. And so if you're, again, if y- if you're sitting here, you're a superintendent or you're a teacher, like, how do you start the question for alignment?
And a challenge, I guess, is that every edtech company, every AI company knows it should market [00:22:00] itself as, "I do everything. I'm aligned to everything," right? HQIM is in every pitch deck now or every sales thing. So- How do you cut through that if you're trying to decide what to adopt?
Emma Dorn: Yeah, I mean, I think it is hard. I think on the curriculum side, Ed Reports is tremendously helpful, but as you said, most things now are Ed Reports green, and so how do you get to the next level of that? There's a few levels of this. The first is just making sure it's meeting the basics. Is it green on Ed Reports?
Is it aligned to standards? Secondly, does it actually specifically state that it's aligned to the underlying curriculum you're using? So if you're using, Eureka Math, is your edtech tool aligned back to Eureka Math? And then thirdly, is there evidence that it's actually working with students? And the evidence for ESSER can help with that.
And yeah, it is difficult. There's no one place where you can get all of those pieces.
Jeremy Singer: Yeah. And would you recommend, I guess it really depends on the technology, can they pilot it with one school or one classroom and see how it [00:23:00] works? Is that a, typical piece of advice you would think about? Or how would you-- are these system-wide kind of implementations often?
Emma Dorn: Yeah, I think there's-- You need to be careful with the piloting versus planning for scale and thinking about, yes, maybe you do end up starting it with one school or a couple of schools and trying a different alternate approaches, but with the mindset that this is to scale, not the mindset that it's a pilot.
I think the danger with pilots is often they can-- you can do 100 pilots and none of them ever scale into anything.
Jeremy Singer: Yeah, I think that's right. And there's also the danger of the learning cycles can take a long time. Yes. It could be years before you can do that. Okay, well, we're gonna go to the fourth lesson, and that's technology can be most powerful when it helps scale interventions that we already know works, things like teacher coaching or structured pedagogy.
Again, this is one that sounds obvious, but and I guess it really bleeds into the conversation we're just having, which is, tell us more about it, give us an example, and [00:24:00] then I, wanna translate it back to here.
Emma Dorn: Yeah. So there are things that we know work in education, and so starting from what we know works and working out how technology can scale some of those things.
in low and middle income countries, we know that structured pedagogy with aligned teacher coaching can be really transformative in improving student outcomes. But the challenge has always been scale. Teacher coaches are expensive. They're hard to attract and retain. And so we have a couple of examples like Sierra in Brazil, the Chisome Project in Kenya, where they've managed to scale teacher coaching, but it's been really hard to replicate that across the continent.
And so if you know something's working, can technology be used to scale the thing that's already working? And so one example of this is Talimabad in, in Pakistan. And what they've done is they've created a WhatsApp native digital coach that supports teacher coaching through text and voice interactions.
So the challenge is, the instructional bottleneck is teachers need coaching, but there aren't enough coaches. Can we do direct AI coaching [00:25:00] in between in-person coaching sessions to continue to support teachers to learn? And it's pretty straightforward. Most teachers have access to a basic smartphone.
They use their phone to record themselves delivering a lesson in the classroom on WhatsApp. They upload the audio file, so it's very much, native to the technology that teachers already have. and then they receive structured feedback across several areas: formative assessment, fidelity to the lesson plan, student engagement, teacher subject knowledge, and effective pedagogical practices.
And they can follow up with this digital coach in a voice debrief and coaching session to get their strengths, their weaknesses, and the strategies to try in, their next lesson.
Jeremy Singer: I should say, you had me at WhatsApp, but, is it all AI-powered or is there humans as well that's giving the coaching?
Emma Dorn: So the actual coaching is AI-powered, but it's trained on a rubric that was created from over 20,000 human coaching classroom observations, and as well as using the World Bank's Teach tool, which is based on, millions of examples of [00:26:00] experiences kind of across the world.
And so what they've been able to do is then train the model such that the AI is actually almost as effective as a human coach in delivering some of the feedback. One-- Really interestingly, one of the initial findings from the AI was that teachers were talking 80% of the time, and so the feedback then was kind of going to teachers to empower students to doing more of the talking.
And then that created its own technological challenge because if you have the students doing more of the talking, then you need to be able to capture different student voices. So it's still very much a, technology that's being developed, but I think really exciting to be able to provide real time. instead of seeing a coach once a year, you could record every lesson if you wanted to.
Jeremy Singer: Right. And what's the evidence of the efficacy of this?
Emma Dorn: So this is earlier. So, realistically, a lot of these AI things are very early. Talemabad's previous program, which included scripted lesson plans, digital teacher training, and in-person coaching, was very effective at improving student outcomes.
It had students gaining an additional one [00:27:00] point five years of learning over a year. And what the WhatsApp teacher coach is doing is hopefully making those gains more scalable, but it hasn't been evaluated in RCTs yet 'cause it's so new.
Jeremy Singer: So you're saying there's evidence that teacher coaching makes sense, and then whether this has replicated the impact of that is TBD.
Right. Exactly. Got it. I buy the argument that, okay, this teacher coaching works, so let's scale-- let's use technology to help scale that. tutoring works or certain forms of tutoring works, let's scale that. Do you worry, though, that these are improvements, but they're not real, like step function changes?
and is there a role for technology to try to be more transformative in some way? Or is, or is-- are we gonna succeed by these kind of incremental improvements in a lot of ways?
Emma Dorn: I mean, I think it's a both/and. You really need both, especially in Sub-Saharan Africa, where nine out of 10 kids are still in learning poverty.
I think you need to do everything you can to scaffold teachers to be able to provide better instruction, but [00:28:00] also do things like the Imagine approach, which is direct to student. And, I think it's in some ways a Band-Aid until you can train up teachers, but there's a whole generation of students in Africa that aren't gonna get taught unless we do something.
So I think you need both.
Jeremy Singer: Yeah, I think that's fair, and I-- let's, as we relate it back to what would be the takeaway for our listeners applying some of these lessons in the US for this, particular lesson, I think it's fair, like the impact, I don't wanna understate the impact. And i-if the current students are at a sort of at a certain level, and maybe that's the lesson in the US, is, "Hey, these can have big gains," these kind of things where the resources just aren't there.
But maybe it's a less valuable for a more resourced system. What would be the takeaway from that if, again, if you're a superintendent here? And then United States Well, I
Emma Dorn: think especially again, where it really ends up making sense is where you have earlier tenure teachers who still do need a lot of teacher coaching and support.
This could be transformative for them. Or in areas where, student outcomes are really particularly [00:29:00] challenged. I think, within the US system, we have maybe not poor to fair, but fair to good to great, right? And so maybe in some of the areas that are challenging, some of these approaches for teacher coaching could be helpful to complement in-person coaching.
I'm not saying it replaces it, but it could complement some of the in-person coaching.
Jeremy Singer: Yeah. And then to be clear, I mean, I think there's a lot of places where US students are way behind, as we talked about, the number of high school students that still need remedial intervention. So I don't wanna act like it's been figured out.
So let's jump to the fifth, and I think it's our final lesson of this conversation, and then we'll, go into broader concepts. The fifth lesson you cheekily say, think beyond the Scantron, that a lot of assessments are designed for system accountability, not necessarily for helping a teacher know what they should do tomorrow.
And I think your argument here is that oral exams can help make that possible. So, so unpack all of this and let's have this discussion.
Emma Dorn: Yeah. I think this one's really exciting, and this one really does have some potential for implications in the [00:30:00] US. This is all based upon an approach called Teach At The Right Level, or TAL.
and these approaches have been very effective at improving basic literacy and numeracy across low and middle income countries. And at the very basic level, it basically means assess kids for where they are. Are they at the point of being a beginning reader? They're reading letters, they're reading words, they're reading paragraphs, they're reading stories.
And then grouping children into those groups to be able to teach them at their level of proximal development. Sounds simple, but often that doesn't actually happen. And historically, some of these learning camps, these TAL learning camps, have doubled the number of kids who could read a paragraph or story.
But the challenge is always the assessment, right? Because the assessment is very time-challenging and requires a lot of expertise to be able to orally assess each child's reading skills. And so Pratham in India is working on an oral reading fluency product, and what it does is it basically the child picks up the cell phone, it reads aloud to the device.
There's a piece [00:31:00] of text on-- from a set item bank on the phone. It works on a $50 cellphone. It's an offline model, doesn't need any internet connection, and it assesses the student's ability to read and provides detailed results to the teacher on overall fluency and common words missed, and then instructional strategies based upon some of those common misunderstandings
Jeremy Singer: That's great.
What have we seen as far as the impact or the evidence behind that?
Emma Dorn: Yeah. So right now, again, it's early, so they don't have as much evidence of impact directly on student outcomes, but it's achieved a 50% time reduction in assessment time. And so that means you can do twice as many assessments, which means that you can iterate more quickly on bringing kids into the right level.
And it's got 98% accuracy in literacy level assignments, and so exciting early results from that.
Jeremy Singer: So the innovation here, we'd all agree that organizing students into similar skills is important or s- similar progressions and effectively evaluating a student is [00:32:00] important, so they need a better acronym than TARL.
But is the, innovation just the idea that this oral exercise is a very effective, scalable way to assess where a student stands at that moment?
Emma Dorn: Yeah, absolutely. And I think, it's partly is working because of, again, how the data was trained, and I think this is like a thread across multiple of these.
This data was trained on over 2,000 hours of children's voices in local dialects and accents, initially in Hindi and Marathi for use in India. And often a lot of the models that exist right now, they're great at recognizing adults' voices, not so much at recognizing children. And so being able to train models really specifically on children's voices to be able to then roll out oral reading assessments.
The Pratham one has mostly been used in their own programs, but there's another oral reading assessment by an organization called Wadhwani, which is kind of a similar idea. It's been trained really specifically in H- Hindi and Gujarati, and it's actually been rolled out across 7.5 million students [00:33:00] in Rajasthan and Gujarat for government remediation efforts.
And so again, reaching large scale really quickly and saving a ton of teacher time, and also in some places enabling facilitators to be able to do this instead of having to have the trained teacher do this very time-consuming evaluation.
Jeremy Singer: I'm just stunned by-- You see seven and a half million, you're just like the, scale of some of these are really impressive and, just hard to fathom.
So the, tool is great for sort of doing the assessment to allow for grouping. You're still doing professional development, I assume, with the teachers to say, "Okay, now that you have this, you have to differentiate instruction according to these groups," et cetera, et cetera. Is the lesson learned for the US just be obviously intentional about there are other tools like that to, to assess and, you can do differentiated instruction that, I've talked to past guests that maybe it's more realistic today than ever before to actually follow through with real differentiated instruction.
Emma Dorn: Yeah, and I think being able to work out, think about assessments beyond the written assessment. I [00:34:00] think some of these oral assessments now are coming online and working out how those can be integrated into your differentiation on a more regular basis so that students really are learning at their proximal level of development.
Jeremy Singer: Yep. Yep. It's a new, tool in the arsenal, no question. I didn't know if we'd get through all five. I'm very happy we got through five case studies. So one thing I'm gonna try to summarize, and, you're gonna tell me how well I did this. So, so across all these examples, there's a common thread that if I were to say that the technology is not the intervention by itself, it only works when it's attached to a clear instructional model, it's aligned to curriculum, it's embedded in teacher routines, it's supported by implementation discipline or high-fidelity implementation.
Is that, the right takeaway? And is that also maybe the lesson learned from the last 25 years of edtech and why it hasn't had the impact we had hoped?
Emma Dorn: No, I think each of those things are tremendously important. starting with the instructional core, starting with what you're actually trying to do, starting with trying to teach [00:35:00] each student at their proximal level of development, but then also starting where you are with the technology you have versus trying to kind of do everything at once.
And so I think a lot of these start with, offline models. They start with things that work in limited connectivity, limited electricity, that are very easy to upgrade teachers' skills to use. And then what's also interesting about these is all of these models have really upgraded teachers' capacities alongside the technology to actually be able to use the technology effectively.
And so that teacher professional development piece is really important.
Jeremy Singer: Yeah. Great. Great. I love that addition, and that's super important. In the show notes, we're gonna attach links to all these studies, so listeners can go nuts and go as deep as, they want. I ask each of my guests a rapid-fire question just to, to get to know you better.
Don't, think too hard on these. What's the one education buzzword you wish we could retire?
Emma Dorn: I would say 21st century skills, but I need to have a caveat that it's not because they're not important. They are. It's that there's this [00:36:00] tendency to think that they're all that's important and forget the ongoing importance of foundational literacy, numeracy, and background knowledge, which I think all are gonna become increasingly important in an AI world.
Jeremy Singer: 100%. What's your favorite book about education or a book that just deeply shaped your thinking?
Emma Dorn: I would say Andreas Schleicher's "World Class," because it's one of the few books that look beyond national borders and really draws lessons from gl-global education systems, and partly just 'cause I'm a PISA data nerd, and I love his analysis of PISA data.
Jeremy Singer: I, I love that. I've not read it, but I will, get it out. one thing that makes you bullish on the future for learners.
Emma Dorn: I think just the demographic dividend in Africa. By 2050, 40% of the world's youth population will be in Africa. Wow. And I'm just excited for the innovation that's gonna come from the continent, the future undiscovered scientists and engineers and poets that are gonna change the world, solve some of our big problems of climate change and political instability and AI, if we can just provide them with the basic [00:37:00] opportunities for education and growth that so many others already have.
Jeremy Singer: Yeah. That's inspiring. And finally, one class, either high school, college, you name it, that you wish all students had to take.
Emma Dorn: So I can think of a couple, but I think I'm gonna go back to basics and say my dream isn't a specific class per se. It's rather that every student would actually leave high school with the foundations that they need to succeed in literacy and numeracy.
Too many students globally aren't getting to high school with those foundations in place, and it feels like high school is really the last chance to build that.
Jeremy Singer: Yeah. And it's so hard to build it that late, but it's still critically important that we, teach these basic skills. So I second that as well.
So okay, now last question. We're five years from now, we, reconvene, we're having a discussion. Of all the things you could imagine have been replicated or imported from international, learnings in the US, what would you think would be the thing that you'd wanna see that would have the biggest impact on [00:38:00] the success of US students?
Emma Dorn: Interesting. I mean, I think it's twofold. I think it's bringing all of the supports to teachers. My sense is it's that teaching and learning is still a very human collaborative endeavor, and that we are not gonna replace-- I hope we're not gonna replace that with a whole bunch of kids looking into little boxes.
And so that we really are able to both scaffold but also reward our teachers effectively for the important work that they do. But the-- in parallel to that, that we've worked out ways of integrating real, truly personalized adaptive learning that every child can move at their own pace and that every child can achieve to their own potential.
Jeremy Singer: That's great. So we're almost supercharging the system as it is today, both helping more tools to help teachers and for students, which would be a nice place for us to reach. Well, listen, thank you so much, Emma. I, know I speak for my listeners as well. We appreciate you expanding the aperture of how we think [00:39:00] about where innovation's happening, what can work, and please keep bringing these examples that you find and these successes around the globe so we can learn not just from within our borders.
So thank you so much for being on Education Equation.
Emma Dorn: You're so welcome.
Jeremy Singer: Thanks for tuning in today. Join the conversation by following the education equation wherever you listen to podcasts.