Keywords

1 Introduction

Virtual worlds are great tools for research, training, entertainment, and education. Even though virtual worlds have served well as an innovative supplementary tool for presenting educational content, students’ learning is influenced by social aspects that virtual environments alone cannot accommodate. To address this problem, virtual agents can take advantage of social affordances to accomplish a learning task [9]. As a result, educational virtual environments that incorporate a virtual agent are more engaging and motivating to learners than the virtual environment alone [8].

Embodied conversational agents (ECAs) [4] are virtual characters that are placed in a virtual environment and may have different functions and different levels of interaction with a user (if any). ECAs serve in a variety of fields, such as entertainment, training, customer service, and education. In education, specifically, ECAs provide supplementary motivation and pedagogy as tutors, learning companions, mentors, tutees, and other pedagogical roles. ECAs whose role is educational are termed Embodied Pedagogical Agents (EPAs) [6]. EPAs can model different instructional roles, such as instructors, mentors, and learning companions. Their goal is, of course, to increase the student’s proficiency in the material being taught by the EPA. Some EPAs have a secondary goal to engage students and in this way to increase their motivation to learn. Strategies to help EPAs achieve engagement have included changing their appearance, their teaching style, and the roles they play within the learning environment [2, 5, 6]. For example, children aged 7 to 11 chose an EPA learning companion over an EPA instructor to teach them multiplication, where children indicated that they could relate more to the learning companion and trusted the EPA [5].

In educational experiences conducted in virtual reality (VR), the user is typically guided by a pedagogical agent as a learning companion (PAL) [9]. While the prevailing consensus is that children tend to prefer agents that look like themselves (e.g., [2, 10]), the effect of the PAL’s gender on learning, rapport, and engagement is an open question. Johnson, DiDonato, and Reisslein [7] reported that children (K-12) preferred an embodied pedagogical agent that represents a young woman close to their age as a virtual learning companion. Conversely, Baylor and Kim [1] found that students showed higher interest when working with a male PAL. Other studies have looked at not only the PAL’s gender but the also preference of the student based on the gender. Haake and Gulz [6], for example, found that female students tended to prefer a task- and relation-oriented communication style for their learning partner. The split in the research results makes it difficult for developers of virtual-reality educational experiences to choose the most effective PAL. Accordingly, our research aims to find which approach is more effective for learners in terms of learning, engagement, and rapport: a PAL that is the same gender as the student, a PAL that is female, or a PAL that is male.

2 The Boston History Experience

We addressed the question of students’ PAL gender preference and effectiveness by comparing learning and rapport outcomes as a function of the gender of a PAL and the gender of the student. To this end, we implemented a multimodal interactive virtual-reality application to teach students about the Boston Massacre in 1770 [12]. In the application, students explore the city of Boston with an assigned PAL that plays the role of a house worker, while the user plays the role of John Adams’s apprentice. In the interaction, the user walks alongside the PAL while having conversations with seven ECAs, representing characters such as Abigail Adams, a tea-shop owner, and a redcoat (see Fig. 1). The interaction between the student and the virtual agents concludes with a conversation with Abigail Adams in which the students explain what they learned and narration by John Adams about events that followed. A final non-interactive cinematic sequence presents additional facts about the events that unfolded after the Boston Massacre.

Fig. 1.
figure 1

Lydia, a pedagogical agent as a learning companion, and a redcoat, in the Boston Massacre history experience (Color figure online)

Full size image

The application is fully functional, and the experience lasts about 20 min. The application was implemented in the Unity game engine, using the UTEP AGENT system [11] and the Microsoft speech recognizer, and was delivered via the HTC Vive VR headset. To assure accuracy, the script was developed in collaboration with a professor of history. Table 1 presents samples of the application’s dialog with some of the virtual agents, with the PAL prompting the student.

For the setting, we created a virtual representation of the central section of Boston in 1770, with dozens of extra agents on the streets (see Fig. 2). The agents representing the characters, including John and Abigail Adams (see Fig. 3), Joshua, a tea-shop owner (see Fig. 4(a)), and Phoebe, a slave (see Fig. 4(b)), speak with recorded voices. Background music was composed and recorded especially for the application by a professional composer.

Fig. 2.
figure 2

Virtual Boston, with dozens of (non-speaking) agents on the streets, was modeled after the streets of Boston in 1770 close to the customs house.

Full size image
Fig. 3.
figure 3

Embodied conversational agents in the roles of John Adams and Abigail Adams.

Full size image
Table 1. Samples of agents’ utterances, with prompts from the PAL.
Full size table
Fig. 4.
figure 4

Embodied conversational agents in the role of (a) Joshua, the owner of tea shop, and (b) Phoebe, a slave.

Full size image

3 Methodology

Our study had two independent variables, the PAL’s gender and the student’s gender. For the first variable, the VR experience had two versions. In one version, the PAL is a female character named Lydia. In the second version, the PAL is a male character named Henry (see Fig. 5). The information material presented to the participants in both versions was the same; the only aspect that differed was the gender of the PAL. Half of the participants interacted with the female agent Lydia, and the other half interacted with the male agent Henry. We hypothesized that:

  • Subjects interacting with the female PAL will perform better in recalling facts

  • Subjects interacting with the female PAL will report higher rapport

  • Subjects interacting with the PAL of the subject’s gender will perform better in recalling facts

  • Subjects interacting with the PAL of the subject’s matching gender will report higher rapport

  • Subjects who report higher rapport with the PAL will perform better in recalling facts

The study used a between-subjects design with 90 participants (69 male and 21 female), all undergraduate students at a public university, who were randomly assigned to one of the four conditions.

Fig. 5.
figure 5

PAL companions Lydia (left) and Henry (right)

Full size image

The study had two dependent variables, rapport and learning. We measured rapport with a post-interaction Likert-scale survey adapted from [3]. Table 2 presents the ten items in the survey, plus an additional open-ended question for qualitative responses. We measured learning with a post-interaction quiz, developed with the help of our consulting professor of history, presented in the appendix. The quiz had ten questions, six of which were multiple choice, two were true or false, and two were fill-in-the-blank.

4 Results

Our first hypothesis was that subjects interacting with the female PAL would perform better in recalling facts. The mean learning scores for students interacting with the male and female PALs were 53.1 and 56.0, respectively, but this difference was not significant (one-tailed t test, p > 0.15). If there is an effect, the effect appears to be small. Figure 6 shows the results for rapport and learning, differentiated by the two independent variables, PAL gender (L[ydia] or H[enry]), and participant gender (F or M). Table 3 reports the values across all conditions.

Table 2. Post-interaction rapport survey questions
Full size table
Fig. 6.
figure 6

Distribution of results (rapport and learning), differentiated by gender of PAL (L[ydia] or H[enry]) and gender of participant (F or M).

Full size image
Table 3. Summary statistics for information recall and rapport, as a function of the gender of agent and participant.
Full size table

Our second hypothesis was that subjects interacting with the female PAL would report higher rapport. There is suggestive evidence that this is true, as can be seen in Fig. 4. The mean rapport scores for students interacting with the male and female PALs were 4.48 and 4.65, respectively. A t-test indicates that this is suggestive of a significant difference (one-tailed t-test, p < 0.08).

Our third hypothesis was that subjects interacting with the PAL of the subject’s gender would perform better in recalling facts. Again, the evidence mildly suggests that this is true. The mean learning scores for same-gender and opposite-gender student-agents pairs were 56.8 and 52.2, respectively. A t-test indicates that this is possibly suggestive of a significant difference (one-tailed t-test, p < 0.10).

Our fourth hypothesis was that subjects interacting with the PAL of the subject’s gender will report higher rapport. Our data suggest that is not true. The mean rapport scores for same-gender and opposite-gender student-agents pairs were 4.53 and 4.58, respectively. A t-test indicates that this difference is not significant (two-tailed t-test, p > 0.67). Indeed, the mean scores were slightly opposite of the expected effect.

Our fifth hypothesis was that subjects who report higher rapport with the PAL will perform better in recalling facts. As suggested by the scatter plot shown in Fig.  7, there appears to be no correlation between learning and rapport. The correlation is less than 0.0003, and R2 is 6.56E-08.

Fig. 7.
figure 7

Scatter plot of results (rapport and learning)

Full size image

Table 4 shows the performance of all participants for each question. Multiple-choice question seven asked student which of four multiple choice responses was false. Correctly, none of the all 90 participants selected as false the response “Slavery existed in Boston at the time of the Massacre.” We speculate that the reason for this being an obvious wrong choice was because the students had interacted with a slave as part of the experience.

Table 4. Percentage of people who scored the correct response for each question in the questionnaire for fact recall. Details of the questions are provided in the appendix.
Full size table

Limitations. This study’s results are subject to two possible limitations. First, the study had a gender imbalance of participants, with more males than females. By the conclusion of the experiment, 35 males and 10 females interacted with PAL companion Henry while 34 males and 11 females interacted with PAL companion Lydia. This distribution reflected the underlying student population of the college in which the participants were enrolled but made less reliable the findings with respect to female participants. Second, the Boston Massacre History Project application was designed for eighth-grade students, but despite agreement from a middle-school principal we were unable to obtain permission from the school district to conduct the study with eight-graders. Consequently, the study’s participants were undergraduate students, whose learning styles and rapport preferences may differ from those of eighth-graders.

5 Conclusion

Our data suggest that female participants with the female PAL learned more than female participants with the male PAL, even though female subjects, on average, reported the same level of rapport regardless of the gender of the PAL. Our data also suggest that male participants learned marginally more with the female PAL than with the male PAL, although the male participants, on average, reported slightly higher rapport with the male PAL. These results suggest that (1) both male and female participants will learn well with a female PAL, and (2) differences in rapport do not seem to affect learning.