Harvard AI Tutor Study: Custom Chatbot Doubled Physics Students’ Learning Gains

The Harvard AI tutor study has drawn attention to how carefully designed artificial intelligence could change the way students learn. The experiment involved 194 students enrolled in Physical Sciences 2, a Harvard course for life sciences majors. Preliminary findings reported in September 2024 suggested that students using a custom AI tutor achieved learning gains about twice those recorded by students in an instructor-led active learning setting.

Students reported stronger engagement

The experiment compared two learning approaches across consecutive weeks. One group initially participated in an instructor-guided active learning lesson, while another used the AI tutor outside class. The groups then switched approaches the following week.

Researchers measured students’ understanding using pre-tests and post-tests and also collected feedback on engagement, enjoyment, motivation and growth mindset.

The preliminary analysis showed substantially higher learning gains among students who worked with the AI tutor. Those students also reported greater engagement and motivation during the learning process.

Importantly, the comparison was not between AI and a conventional lecture. Harvard’s classroom approach already incorporated active learning and research-backed teaching techniques, making the findings particularly notable.

The AI was designed like a tutor

The system tested by the researchers was not simply an unrestricted chatbot. Faculty designed detailed instructions intended to make the AI behave more like an experienced instructor.

The system used the GPT application programming interface alongside pre-vetted conversations, content-rich prompts and structured feedback. Its design focused on personalised guidance and allowing students to learn at their own pace.

Students who already understood a concept could progress faster, while those struggling could spend additional time asking questions and working through problems.

This approach could also influence how classroom time is used. If students can receive personalised explanations outside class, teachers could potentially dedicate more classroom time to advanced problem-solving, projects and collaborative activities.

Researchers caution against replacing teachers

The findings do not suggest that AI should replace educators. The researchers stressed that poorly designed AI tools could undermine learning if students rely on them to think instead of developing their own reasoning skills.

The experiment instead points toward a model in which AI supports established teaching methods through personalised feedback and guided practice.

Further pilots were being considered for subjects including multivariable calculus and other introductory courses. The final research publication was still pending when the preliminary findings were reported, so the results should be interpreted as early evidence rather than a definitive conclusion about AI tutoring.