Harvard Students Build Their Own AI Game to Learn a New Language

Learning a new language demands a lot of practice and real-life interactions.

But what if you could use AI to help? That's exactly what Harvard faculty member Nicole Mills set out to do with a beginning language class at Harvard University using Wonda.

The experiment was a first-of-its-kind simulation exercise, in which 50 students were invited to participate in their own murder mystery investigation using AI as part of their French language class.

With Wonda's latest AI capabilities, the students were able to create characters and participate in a murder mystery investigation in just two 75 minute classroom sessions and practice their pronunciation, writing, and conversational skills in a highly engaging and culturally immersive virtual environment.
‍
As a starter, here is a 6' presentation of what students managed to create and how they could interact with their own AI characters in class.

‍The Mystery Game

The experiment was part of a beginning language course where students engage in a global simulation of life in Paris. Students create fictional French-speaking characters who live virtually in the same building in Paris and create a collective storyline by writing memoirs from their character’s perspective. ‍

At the end of the semester, during a unit on Fine Arts in Paris, they learn that one of the characters was murdered in a surrealist murder scene in the Centre Pompidou, one of the city's most well-known modern art museums. ‍

In previous years, students would submit a 250-word composition in French describing their alibi, potential suspects, and thoughts about who committed this surrealist crime and later participate in an in-class "trial" where they interrogated the characters to unveil the guilty neighbour. ‍

But this year, Nicole wanted to do it differently using AI as part of Wonda Summer 2023.

How Students Created their Own AI Game

Using Wonda latest AI capabilities, students were able to create their own personalized 3D animated avatars to represent their fictional character.

With a few simple clicks, they designed their character’s own avatar by selecting from a wide variety of facial features, hairstyles, and clothing options using ReadyPlayerMe and Wonda. ‍

The operation was completed in just 10 minutes. Using a simple copy-paste, students added the avatar to the pre-designed virtual environment created by Rev(e) studio. In this case, the virtual environment was a gallery space in the Centre Pompidou museum including walls adorned with multiple adaptations of Magritte’s surrealist work “Querelle des Universaux” and window views of Paris (all created with AI tools!).‍

To power each avatar with stories about the character, each student placed their compositions and written work from in-class activities including their self-portraits, alibis and questions regarding the murder mystery in a prompt for their AI character using a simple copy-paste. ‍

Students could also choose a voice for their character from multiple options. Following some initial class discussion, the students then voted on the two most guilty suspects. ‍

The second classroom session was dedicated to the interrogation. All students were gathered in a room in front of a large screen and the instructor launched the immersive experience from their computer.

Once launched, the students were able to view all of their characters in the virtual gallery space, with particular attention to the two guilty suspects, key evidence, and the victim. 

After orienting students to the gallery space, each student was able to see their own character come to life. Each student had the opportunity to ask their own character a personalized question and anxiously wait to see how their character might respond!

‍
After the character introductions, the interrogation of the two guilty suspects began.

Students were asked to carefully prepare their questions using targeted grammatical structures and practice their pronunciation to ensure that their questions would be understood by their characters. By simply clicking on a character and hitting the microphone icon, student questions were registered.

The answer was transcribed automatically by a Speech AI Model and communicated to ChatGPT, which generated an answer based on the previously posted student written prompts for their characters. ‍

Each character's response was first shown in a written chat box and then instantly generated into a voice response using the student’s selected voice for their character. After multiple Q&As with the suspects, the students then voted to choose the guilty party before they heard the truth from the very point of view of the victim’s ghost - who magically comes to life at the end!

Achieving Optimal Engagement

The results were outstanding in terms of engagement. Students expressed strong emotions and reactions during the interrogation, such as laughing in groups at surprising and unexpected character responses.

"We could start earlier in the semester!"
Harvard Undergraduate student‍

When asked about their classroom experience at the end of the two sessions, students were overwhelmingly positive, with 100% of them stating that it was a highly engaging experiment, citing strong indicators of behavioral, cognitive, social, and emotional engagement.

Most of all, the fact that the AI would sometimes not understand their questions due to mispronunciation was a strong motivator for students to make more efforts to work on their pronunciation. ‍

Overall, the project integrated attention to grammatical accuracy, comprehension of both written and audio texts, and writing skills in a simulated, yet culturally grounded context.‍‍

For an in-depth presentation of the project, here is the 75' Webinar recorded at Harvard Language Center with Nicole Mills:

What’s next?

As artificial intelligence capabilities expand and develop, educational research advocates for "intelligence augmentation" so that instructors may collaborate effectively with AI to cultivate creative, communication, collaboration and critical thinking skills.

“The way Wonda integrated generative AI to simulate conversations is an important step for experiential learning" Chris Dede, Senior Research Fellow at Harvard Graduate School of Education

With this goal in sight, innovative pedagogical applications of AI in the teaching of languages and cultures can offer language instructors models to differentiate instruction and ultimately prepare language learners for tomorrow's world.

Following up with this project, Wonda launched an AI pilot program in the Summer 2023 to which 7 institutions participated to further explore the possible applications of its new capabilities in language learning but also in other areas such as:

• Immersive Cultural Education: Wonda AI Characters can be used to provide immersive educational experiences that allow students to learn about different cultures and create and play games that take place in historical or geographical contexts, such as the Renaissance or the Amazon rainforest.
  
• Professional Training and Development: Wonda AI Characters can be used to train professionals in various fields. For example, healthcare workers can create and play games that simulate patient interactions scenarios, allowing them to practice and improve their skills in a low-pressure environment.
  
• Design Thinking Workshops: Wonda AI Characters be used to facilitate virtual team building exercises for remote teams to brainstorm on new ideas. For example, teams could create and play games that require collaboration and problem-solving, allowing them to build stronger relationships and improve their communication and teamwork skills while finding new ideas for their projects.

“GPT by itself is an amazing tool, but coupled with Wonda, it sparks the imagination of what you can do with AI ”
Anoop Bal, Product Designer at Rev(e) Studio