Kari L. Visscher, Nicholas Woolridge, Jodie Jenkinson, and Kevin C. Kain

Download PDF

The objectives of this study were to investigate the quantity and quality of undergraduate medical students’ learning from a three-dimensional animation, and to evaluate whether timeline pacing used as a post-production cueing strategy can improve the educational effectiveness of the animation. First- and second-year undergraduate medical students (n = 89) at the University of Toronto evaluated a three-dimensional animation depicting the interaction of Plasmodium falciparum malaria with HIV-1 within the placenta. The students were assigned by means of permuted block randomization to view the animation either with timeline pacing (edited version) or without (unedited version). The double-blinded evaluation consisted of the students’ completion of a randomized computer generated pre- and post-test and a feedback form. To ensure that all participants had similar baseline knowledge of the subject, the results of participants who scored 30% or above on the pre-test (n = 7) were omitted. Mean test scores for students in both groups improved after they watched the animation only once (unedited version: t39 = 23.0, p ≤ 0.001; edited version: t41 = 24.7, p ≤ 0.001). However, no significant difference in learning was detected between the two versions of the animation (t80 = 0.762, p ≤ 0.230). For both groups, the mean difference between scores for pre-and post-testing of recall questions was significantly higher than that for testing the application questions (unedited version: t39 = 8.23, p ≤ 0.001; edited version: t41 = 9.09, p ≤ 0.001).A complex three-dimensional animation aided first- and second-year undergraduate medical students’ learning, particularly learning factual knowledge. Timeline pacing did not have an impact on the learning outcome.

Figure 1. Selected frames from the animation by Kari Visscher entitled “Malaria, HIV-1, and Pregnancy: A Cycle of Affliction.” Images are from the opening sequence (a), second sequence (b), third sequence (c–g), and closing sequence (h).

Figure 2. Example of recall and application multiple choice questions used in the evaluation.

Figure 3. CONSORT flow diagram of the progress through the phases of this randomized trial.

Figure 4. Box (median and interquartile range [IQR]) and whisker (range) plots showing test scores (%) on the pre-test and post-test for the unedited version (light grey) and the edited version (dark grey) of the animation. Grey dots represent outliers.

Figure 5. Box (median and IQR) and whisker (range) plots showing test scores (%) on the recall questions of the pre-test and post-test for the unedited version group (light grey) and the edited version group (dark grey). Grey dots represent outliers.

Figure 6. Box (median and IQR) and whisker (range) plots showing test scores (%) on the application questions of the pre-test and post-test for the unedited version group (light grey) and the edited version group (dark grey). Grey dots represent outliers.

Introduction

Over the past decade, the use of three-dimensional (3D) animations in higher learning has been on the increase (Azer 2007; Mayer and Moreno 2002), and, like most educational material, the effectiveness of these teaching aids depends heavily on the quality of their design (Lowe 2004). Since animations have the potential to distract students from the intended message (Choules 2007), designers need to actively manipulate the way content is presented to guide students to relevant information.

This manipulation, or cueing of content, although useful in all types of instruction, is particularly important to the design of instructional 3D animations because of their linear, often non-interactive nature, which makes controlling the rate of learning in classroom settings difficult. Current types of successful cueing techniques used by animation designers include narration (Mayer and Anderson 1991, 1992); overt changes in the characteristics of the focal objects such as changes in form (transformation), position (translation), and inclusion (transition) (Lowe 2003; Rensink 2000); and surface structure such as colour, texture, and level of detail (Weiss et al. 2002; Wilson-Pauwels 1997).

Studies (Bolls and Muehling 2003; Lang et al. 2000, 2003) evaluating the effects of the design of television news and commercials on viewers’ attention and comprehension suggest that the use of post-production cues such as pacing are effective. In particular, this research (Bolls and Muehling 2003; Lang et al. 1999, 2000, 2003) looked at altering pace, which is the speed at which visual information is presented to the viewer for cognitive processing. From this research, Potter and Lang (1999, 2001) devised a set of seven post-production guidelines to make stories easier to understand. One of these guidelines suggests the use of slow-to-moderate pacing, or timeline pacing, for complex stories. Timeline pacing, the speed at which the linear representation, or timeline, of a film or animation is viewed, is a standard post-production editing technique used in cinematic narrative, particularly in the action-adventure genre (Peebles 2004). However, the effectiveness of timeline pacing in other dynamic media such as in instructional animation has not been investigated.

For this study, a complex 3D animation was used to investigate the quantity and quality of undergraduate medical students’ learning from 3D animation, and to evaluate whether timeline pacing used as a post-production cueing strategy could improve the educational effectiveness of the animation. For the primary outcomes, it was hypothesized that the students would learn from the animation and achieve significantly higher post-test than pre-test scores, and that students exposed to the animation with timeline pacing used as a cueing strategy would achieve a larger mean difference in pre- and post-test scores than the group viewing the animation without the timeline pacing.  For the secondary outcomes, it was hypothesized that the students would perform better on recall questions than on application questions and that this difference would be greater for the group viewing the edited version of the animation.

Methods

Participants

A convenience sample of 89 first- and second-year undergraduate medical students attending the courses Brain and Behavior (BRB 111S) and Foundations of Medical Practice (FMP 211S) at the University of Toronto was recruited by the primary researcher between May 1 and May 10, 2006. According to the Central Limit Theorem, a minimum sample size of 30 students was required for each of the two treatment groups (Fowler et al. 1998).

Undergraduate medical students were chosen as the target audience because of their general understanding of embryology, anatomy, HIV-1 and malaria, which gave them the necessary level of education to participate in the study. The pre-test was used to test prior knowledge specific to the content presented in the animation and the results of participants who scored 30% or above (n = 7) were omitted. Participation was voluntary. To provide an incentive to participate in the study, participants were entered in a draw for an Apple iPod Shuffle (Apple Inc., Cupertino, CA). The study was approved by the University of Toronto Ethics Review Office (protocol reference # 17211), and participants gave written, informed consent before participating in the study.  

Creation of the animation           

The animation created for the study explained the complex series of events that cause pregnancy-associated malaria and the interaction between malaria (Plasmodium falciparum) and HIV-1 within the placenta. The content expert, Dr. Kevin C. Kain, ensured that the information presented was medically accurate and complete.

The resulting five-minute, fully narrated animation called “Malaria, HIV-1, and Pregnancy: A Cycle of Affliction,” created by Biomedical Communication’s graduate Kari Visscher, began with silhouetted video footage of a pregnant woman in African dress being bitten by a mosquito (Figure 1a), which set the context for the animation. The second sequence used two-dimensional animation to introduce the baby in vivo and the physiology of the normal placenta (Figure 1b). The third sequence used 3D animation to describe the events of pregnancy-associated malaria and the hypothesized role of monocyte interaction in increasing the HIV-1 viral load within the placenta (Figures 1c–g). The closing sequence used video footage similar to that shown in the introduction to describe how this interaction between malaria, monocytes, and HIV-1 could have an impact on the mother and child (Figure 1h).

To test the effectiveness of timeline pacing as a post-production cueing strategy, two versions of the animation were created. In one version (the unedited version), the timeline was not altered. In the second version (the edited version), the timeline was paused (freeze-framed) for three seconds at the seven key points that presented information essential to understanding the complete story to assess whether timeline pacing could guide learners’ attention during instruction and thus increase their ability to recall highly relevant information. The narration was identical for both versions and corresponded to the visual events in the narrative.

The video was edited with QuickTime Pro 7™(Apple Inc, Cupertino, CA) and Adobe After Effects 6.5™ (Adobe Systems Inc., San Jose, CA). The narration was edited in BIAS Peak 3™ (Berkley Integrated Audio Software Inc., Petaluma, CA). The two-dimensional animation was created in Adobe Illustrator CS2™ (Adobe Systems Inc., San Jose, CA ) and Adobe Flash MX 2004™ (Adobe Systems Inc., San Jose, CA). The 3D animation was created in MAXON Cinema 4D R9™ (MAXON Computer Inc., Newbury Park, CA), and the entire piece was composited and edited in Adobe After Effects 6.5™ (Adobe Systems Inc., San Jose, CA).

Evaluation

One second-year medical student and a graduate student, both of whom had expert knowledge about the content of the animation, served as pilot subjects to validate the materials (the two versions of the animation), establish the quality of the testing instruments (pre- and post-tests and feedback form), and determine an appropriate length of time for the evaluation.

Study participants were asked to sign up for one of the five predetermined testing dates and times. Using permuted block randomization, the primary researcher randomly assigned the animation versions to the testing dates so that about half of the participants would evaluate the edited version and half, the unedited version.  The evaluations took place in the Microcomputer lab and Biomedical Communications computer lab in the Medical Sciences Building at the University of Toronto.

Each participant was assigned to a computer with the online evaluation (designed in Macromedia Flash MX™) enabled. Random number ID’s were generated and each participant was assigned a unique ID generated by the Flash program. Answers to the online test questions were automatically recorded to a secure MySQL™ database server (MySQL Inc., Seattle, WA). Two trained volunteers who were blinded to the group assignment were present during the evaluations to answer questions and play the animation.

Each participant completed the five steps of the evaluation: granting informed consent, completion of the pre-test, viewing the animation (without user control), completing the feedback form and completing the post-test. Participants were given unlimited time to provide their written consent, then about five minutes to complete each of the final four sections, although extra time was allotted if necessary. 

The pre- and post-tests (tests A and B) contained 13 similar multiple-choice questions (nine factual and four application questions). The multiple-choice format consisted of one answer, two distracters, and one “Don’t know” option to decrease the amount of guessing on the test and to increase the test’s reliability, as recommended by Maxwell (1994). Only one question appeared on the screen at a time. To prevent participants looking for answers to test questions from within the questions themselves, participants were unable to return to previous questions once they had answered them. 

The factual questions tested for participants’ ability to recall the material being taught, and the application questions tested for their ability to use this material in new situations (for sample test questions, see Figure 2).  To ensure consistency in the level of difficulty of the pre- and post-tests, the Flash module randomly assigned half of the participants to receive test A as the pre-test and test B as the post-test, and the other half to receive test B as the pre-test and test A as the post-test.

Data analysis

A Student’s paired t-test was used to determine whether participants achieved significantly higher post-test than pre-test scores after viewing the animation. A Student’s independent t-test was also used to measure differences in post-test scores for participants viewing the edited and unedited versions of the animation. Since multiple independent tests were completed, Bonferroni’s correction was used to correct the alpha level to 0.017 to decrease the probability of a type I error.

Bonferroni’s correction was also used to analyze whether (1) participants achieved higher post-test than pre-test scores for the recall questions for each version of the animation, (2) the mean difference in test scores for the recall questions for the edited version were higher than that for the recall questions for the unedited version, (3) the participants achieved higher post-test than pre-test scores for the application questions for each version, (4) the mean difference in test scores for the application questions for the edited version was higher than that for the application questions for the unedited version and (5) the mean difference in recall scores was higher than that for the application scores for each version.

To clarify how the statistics are meant to be read, “unedited version: t39 = 23.0, p ≤ 0.001” means that a Student’s paired t-test was used to compare the marks of 40 students who completed the pre- and post-tests before and after watching the unedited version of the animation. The number 39 is the degrees of freedom (n-1) used for this calculation, and since p ≤ 0.01 we are 99% confident that the difference detected is not due to chance alone.

To see the animation and a visual explanation of the study please visit www.bmc.med.utoronto.ca/~kari and click on the “Master’s research” link. 

Results

Figure 3 summarizes the number of students who completed the study. A total of 89 first- and second-year medical students were enrolled in the study; however, results for seven students were excluded from the analysis because their pre-test scores were >30%.

For both versions of the animation, participants had significantly higher mean post-test then pre-test scores (unedited version: t39 = 23.0, p ≤ 0.001; edited version: t41 = 24.7, p≤ 0.001). The post-test scores for the edited and unedited versions were not significantly different (t80 = 0.762, p ≤ 0.230) (Figure 4).

For both groups, participants had significantly higher mean post-test scores than pre-test scores for the recall questions (unedited version: t39 = 22.0, p ≤ 0.001; edited version: t41 = 25, p ≤ 0.001). Scores for the edited version were not significantly different from those for the unedited version (t75 = 1.084, p ≤ 0.141) (Figure 5).

For both groups, participants had significantly higher mean post-test scores than pre-test scores for the application questions (unedited version: t39 = 5.15, p ≤ 0.001; edited version: t41 = 6.32, p ≤ 0.001). Mean post-test scores for the edited version were not significantly different from those for unedited version (t78 = –0.327, p ≤ 0.628) (Figure 6).

For both the edited and unedited versions, the mean difference in scores for the recall questions was significantly higher than the mean difference in scores for the application questions (unedited version: t39 = 8.23, p ≤ 0.001; edited version: t41 = 9.09, p ≤ 0.001) (Figures 5 and 6).

Discussion

The results of our study suggest that since the first- and second-year undergraduate medical students achieved significantly higher post-test than pre-test scores (65-70% increase) after viewing either version of the animation, they were able to learn the content presented (Figure 4). The complex animation did not seem to overload working memory, which is a major concern for instructional material that involves multiple-channel systems (visual channels, including pictorial and textual images, and auditory channels) (Moore et al. 2004; Anglin et al. 2004).

The amount of learning also did not seem to be affected by the use of timeline pacing as a cueing strategy. Since cueing strategies are designed to help overcome the limitations of working-memory capacity by loosely chunking information (Bruer 2003), this neutral result may reflect our finding that working-memory was not overburdened by the content presented in the animation.

These results highlight the importance of knowing the characteristics and capabilities of the target audience for which an instructional animation is designed (Newby et al. 2006). For example, in the current study, the positive feedback received during the evaluation and high turnout suggests that the participants were motivated to learn the subject matter. Also, because the target audience was medical students, it is reasonable to assume that the participants were skilled, efficient learners. Consequently, this target audience may not have been sensitive to timeline pacing as a cueing strategy because they would be able to learn from virtually any well-designed instructional medium as long as they were motivated and given enough time to process the information (Lumsden et al. 2005). Future research that tests timeline pacing on a variety of target audiences may more accurately reveal the usefulness of this cueing strategy in instructional animations.                       

In addition to the primary outcomes both groups performed significantly better on recall questions than on application questions (Figures 5 and 6), suggesting that the instructional animation engaged the medical students primarily in rote learning (filling the memory with new factual knowledge), as opposed to skill learning (learning a new skill) or constructivist learning (constructing new and meaningful models that integrate prior knowledge) (Mayer 2003). This result may have implications for methods of integrating instructional animations into the existing medical curriculum. For example, an animation could be made available before a corresponding lecture so that the medical students could learn the appropriate, accurate facts in a relatively short amount of time.

The limitations of this study include a small to moderate sample size and as such, the results may not be completely generalizable to the broader medical student body. Also, selection bias would be a factor since the student’s volunteered to participate. Furthermore, since the participants were not retested after a longer period of time (weeks to months) it is uncertain that the learning of new factual knowledge is permanent or merely a transient phenomena.

Conclusions

In the current study, after viewing a five-minute animation once the undergraduate medical students were able to increase their factual knowledge of the topic by approximately 65–70% (Figures 4 and 5). This finding may have implications for how to best integrate complex animations into higher education.

The use of timeline pacing as a post-production cueing strategy had no effect on learning (Figure 4). However, the true effectiveness of this cueing strategy may not have been properly investigated because of masking factors such as the target audience’s learning expertise and level of interest in the lesson material.

Acknowledgments

This study was funded in part by an Ontario Graduate Scholarship and by the Vesalius Trust Alan W. Cole Scholarship. This paper was prepared with the assistance of Sharon Nancekivell, medical editor, Guelph, Ontario, Canada. The authors would also like to thank John L. Semple, Charles J. Lumsden, Brian Sutherland, Marc Dryer, Linda Wilson-Pauwels, and Patricia Stewart for their contributions to this work.

References

Anglin, G. J., Vaez, H., & Cunningham, K. L. 2004. Visual representations and learning: the role of static and animated graphics. In D. H. Johanssen(ed), Handbook of research on educational communications and technology. (pp.865-917).New Jersey: Lawrence Erlbaum Associates, Inc.

Azer, S.A. 2007. Medical education at the crossroads: which way forward? Annals of Saudi medicine 27(3):153-157.

Bolls, P. D., &. Muehling, D. D. 2003. The effects of television commercial pacing on viewers’ attention and memory. Journal of Marketing Communications 9:17–28.

Bruer, J. T. 2003. Learning and technology: A view from cognitive science. In H. F. O’Neil, Jr., and R. S. Perez (eds), Technology Applications in Education:  A Learning View. (pp. 159-172). New Jersey: Lawrence Erlbaum Associates, Inc.  

Choules, A.P. 2007. The use of e-learning in medical education: a review of the current situation. Postgraduate medical journal 83(978):212-216.

Lang, A., Potter, R. F., Bolls, P. D. 1999. Something for nothing: Is visual encoding automatic? Media Psychology 1(2):145-163.

Lang, A., Potter, D., Grabe, M. E. 2003. Making news memorable: Applying theory to the production of local television news. Journal of Broadcasting & Electronic Media 47(1):113-123.

Lang, A., Zhou, S., Schwartz, N., Bolls, P. D., Potter, R. F.(200). The effects of edits on arousal, attention, and memory for television messages: When an edit is an edit can an edit be to much? Journal of Broadcasting & Electronic Media 44:94-109. 

Lowe, R. 2004. Interrogation of a dynamic visualization during learning. Learning and Instruction 14:257-274.

Lowe, R. 2003. Animation and learning: selective processing of information in dynamic graphics. Learning and Instruction 13:157-176.

Lumsden, M. A., Bore, M., Millar, K., Jack, R., Powis, D. 2005. Assessment of personal qualities in relation to admission to medical school. Medical Education 39:258-265.

Mayer, R. E., Anderson, R. B. 1992. The instructive animation: helping students build connections between words and pictures in multimedia learning. Journal of Educational Psychology 84(4):444-452.

Mayer, R. E., Anderson, R. B. 1991. Animations need narrations: an experimental test of dual-coding hypothesis. Journal of Educational Psychology 83(4):484-490.

Mayer, R. E., Moreno, R. 2002. Animation as an aid to multimedia learning. Educational Psychology Review 14(1):87-99.
           
Mayer, R. E. 2003. Theories of learning and their application to technology. In H. F. O’Neil, Jr., and R. S. Perez (eds), Technology Applications In Education:  A Learning View. (pp. 127-158). New Jersey: Lawrence Erlbaum Associates, Inc.   

Moore, D. M., Burton, J. K., Myers, R. J. 2004. Multiple-channel communication: The theoretical and research foundations of multimedia. In D. H. Johanssen (ed), Handbook of research on educational communications and technology. (pp.981-1006). New Jersey: Lawrence Erlbaum Associates.

Newby, T. J., Stepich, D. A., Lehman, J. D., Russell, J. D. 2006. Educational Technology for Teaching and Learning 3rd ed. New Jersey: Pearson Education, Inc.

Peebles, S. 2004. Gunning for a new slow motion: The 45-degree shutter and the representation of violence. Journal of Film and Video 56(2):45-54.

Potter, D., Lang, A. 1999. The seven habits of highly effective storytellers. Communicator October:54-59.

Potter, D., Lang, A. 2001. Bridging the gap: Applying the lessons of research in TV newsrooms. Electronic News: A Journal of Applied Research and Ideas 11:1-5.

Rensink, R. A. 2000. Visual search for change: a probe into the nature of attentional processing. Visual Cognition 7(1/2/3):345-376.

Weiss, R. E., Knowlton, D. S., Morrison, G. R. 2002. Principles for using animation in computer-based instruction: theoretical heuristics for effective design. Computers in Human Behavior 18:465-477.

Wilson-Pauwels, L. 1997. Bringing it into focus: visual cues and their role in directing
attention. The Journal of Biocommunication 24(3):12–16.

Authors

Kari Visscher BSc, MScBMC, MD(c) is a graduate from the Master’s in Biomedical Communications at the University of Toronto and is currently completing a medical degree at the University of Toronto. kari.visscher@utoronto.ca

Nicholas Woolridge  BFA, BScBMC, MSc is currently a tenured Associate Professor in the Biomedical Communications. He conducts research in the development of digital media as instruments of biomedical research, teaching, and patient assistance.

Jodie Jenkinson  BA, MSc.BM is an Assistant Professor of Biomedical Communications where she teaches Community-Centred Design Research, New Media Design and Introduction to Information and Data Visualization (co-taught with Nick Woolridge). Her research interests include information aesthetics, visual literacy, and the design and evaluation of media-based health and science communication tools.

Kevin C. Kain  MD, FRCPC is a Professor of Medicine, University of Toronto, the Director of the McLaughlin-Rotman Centre for Global Health, holds a Canada Research Chair in Molecular Parasitology, and is the Director of The Center for Travel and Tropical Medicine at The Toronto General Hospital. He is a Clinician-scientist at the University of Toronto and his research efforts focus on 1. Global Health; 2. malaria, and 3. diagnostics and surveillance of emerging infectious diseases associated with travel, migration and blood safety.