Jodie Jenkinson, B.A., M.Sc.BMC,
Nicholas Woolridge, B.F.A., B.Sc.AAM, M.Sc.,
Linda Wilson-Pauwels, A.O.C.A., B.Sc.AAM, M.Ed., Ed.D.,
David McCready, MD FRCSC,
Meaghan Brierley, B.F.A., P.G.Dip. Medical Photography, M.Sc.BMC
The Interpretive Visualization (IVIS) Group at the Division of Biomedical Communications, Dept. of Surgery, University of Toronto has developed a visually-oriented, Internet-based Continuing Professional Development (CPD) course on Sentinel Node Biopsy. The site design methodology involved an extensive needs assessment, iterative formative evaluations of site and media design, a summative evaluation of the project, and a final evaluation for certification. Special emphasis was placed on asynchronous Web-based evaluation of the visual media-including still images, animations, and interactive figures-used in the course. Results reinforced the importance of: needs assessment; a user-centered design process; and rapid prototyping.
Introduction
Biomedical Communicators have long been sensitive to the perceived needs of their target audiences, but have rarely done extensive evaluation of their work to validate the many conceptual and aesthetic choices they make. In the last three decades, research methodologies have been developed in a number of sciences-including cognitive psychology, human-computer interaction, and ethnology-that can be adapted to the evaluation needs of biocommunicators (those both academically situated and those in the field). The Interpretive Visualization (IVIS) Group at The University of Toronto received research funding to apply some of these methodologies to the development and evaluation of two Web sites on the topic of breast cancer: one targeted toward women at higher risk for breast cancer; and one aimed at physicians seeking information on a new technique used in the surgical management of early stage breast cancer. This paper will report on development and evaluation work by the IVIS group in the latter project, which resulted in a Royal College of Physicians and Surgeons of Canada (RCPSC) accredited Continuous Professional Development (CPD) course in sentinel node biopsy.
The health-care environment is ever-changing. Drivers of this change include new medical technologies, new pharmaceuticals and procedures and new approaches to the delivery of health care, including an increased emphasis on disease prevention, health promotion and informed consent. This environment is creating an increased demand for health communication material that meets the needs of specific audiences, from the lay public to health professionals, sometimes targeting finely partitioned subsections of those audiences. It is vital for biocommunicators to carefully evaluate for whom the material is being created, as well as what the message is, and how the message is to be communicated. Hugo (2000) states that if there is a close link between a particular message design and the profile of the individual or community we aim our messages at, the communicative effect of whatever medium we use will be improved, "A systematic approach to instructional or communication design requires proper media selection that is related to program outcomes." (Tones 1993 and Maibach 1995).
Biocommunicators have traditionally used a practice-based set of heuristics, passed on through mentorship and training, to decide how to approach a particular communications problem. It is possible to be successful using such an approach, but it is also possible to fail. Methods derived from other media development fields may help to make failure less likely.
Research in human-computer interaction and interactive media development shows that successful development methods usually employ the following important steps:
- Needs assessment, where the target users are afforded an opportunity to explain their level of knowledge, work patterns and information and functional needs. These needs inform the development of the project's goals;
- Formative evaluation, where the target users are exposed to the development project in process and afforded an opportunity to influence its direction. This is tightly tied to the next step;
- Iterative design, where formative evaluation feeds back into the project development to inform further design iterations, which are subsequently tested; and
- Summative evaluation, where the completed project is validated against the development goals.
Importantly for media developers, these steps need not involve the population numbers normally associated with statistical significance in biomedical research. In the development of interactive media, it is more effective to conduct formative evaluations with smaller groups across many tests than to conduct a single evaluation with a much larger sample (Nielsen and Landauer 1993). This standard approach to media design also comes with usability benefits; as Preece (1993) writes, "A well-designed user interface makes it easy and natural for a user to break down (or decompose) a task into subtasks and map them on to the system's functions. A poorly-designed computer system requires its user to decompose tasks in unnatural ways and the ensuing mapping is then prone to errors."
Another major concern of ours was the evaluation of the design of the media objects themselves. A number of researchers have made attempts to objectively evaluate the efficacy of various styles of representation, but these have been marked by a notable lack of control over the quality of the stimuli (Peeck 1987 and Dwyer 1972). A well-cited example in contrast to these, in terms of the stimuli quality, would be Hirsch and McConathy (1986), who evaluated thoracic surgeons' subjective preferences in the depiction of surgical procedures. The present paper will present subjective evaluations of some of the media objects designed for this site.
Sentinel node biopsy (SNB) is a relatively recent innovation in breast and skin cancer therapy (McCready et al 1999). The current "gold standard" of treatment for breast cancer is mastectomy or lumpectomy with axillary dissection (Cantin et al 2001). Axillary dissection involves the removal of lymph nodes in the axilla that would usually drain the breast, and would therefore be likely sites for early metastasis. A frequent side effect of axillary dissection is lymphedema in the arm, a condition that leads to considerable patient morbidity. In SNB, the surgeon maps lymph drainage to the so-called "sentinel node(s)," which is/are subsequently removed; if the node(s) tests negative, axillary lymph nodes may be preserved (McCready et al 1999). SNB is currently an investigative approach that promises to reduce patient morbidity while maintaining the success rate of traditional approaches.
Dr. David McCready, of The University of Toronto, agreed to participate in the development of this RCPSC credit course that addresses specific issues in Breast Cancer Surgery with a focus on the theoretical concepts of lymphatic mapping and sentinel node biopsy.
Our goal was to use the principles of participatory design in the development and evaluation of course material that: reflects the professional behavior of surgeons; is Web deliverable; is multiplatform; addresses bandwidth limitations; and is rich in visual didactic content.
Development Stage 1. Needs Assessment of surgeons and physicians
Methods
The initial questionnaire was designed for distribution to 350 surgeons in the Ontario Surgical Oncology Network. The questionnaire assessed the respondent's technology literacy, their preferences and experience with CPD courses and their perceived information needs in the topic area.
Results
Eighty-five surgeons responded by mail and 58 of these respondents (69 percent) agreed to continue with on-line evaluations during site development. Ninety-four percent of respondents had participated in CPD courses. Although, only 13 percent of respondents had completed Web-based CPD courses, 88 percent responded that they would consider taking a Web-based CPD course and 97 percent had consistent access to the Internet.
Respondents identified what is most useful in a CPD course: the need for new (22 responses) and clinically relevant (12 responses) material. Eleven felt that a two-way interaction with colleagues was important. It became very apparent that surgeons were not about to spend time in a program that did not offer them these criteria.
The respondents identified that their knowledge base was weak in the following areas: Chemotherapy (57 percent), tram flap reconstruction (56 percent), breast reconstruction with implant (55 percent), choosing patients for SNB (49 percent), specialized equipment (25 percent) and radioactive tracers used in performing SNB (22 percent). From this Needs Assessment, the IVIS team began to determine the content to be included in this course and design a framework for the user interface.
Development Stage 2. Formative evaluation of visual media
Methods
A set of educational objectives was developed, based on the needs assessment and on input from the course director. Development of visual material was informed by these educational objectives. For example, one objective of the course was to identify key anatomical landmarks essential to axillary dissection. As the educational goal in this instance was to communicate spatial relationships, a 3-D model of the axilla was created for this purpose.
The media was developed using a number of technologies. Subdivision surface-based three-dimensional models of the relevant anatomy were developed and rendered in the Cinema 4D XL 7.3 environment (Maxon Computer 2001). A non-photorealistic rendering method was developed based on work by Hooimeijer (2001); this allowed the creation of media that could be scaled for image complexity. Digital video was assembled and edited using Adobe After Effects 5.5 (Adobe Systems 2001) and compressed using Discreet Cleaner 5 (Discreet 2001). Still images were edited using Adobe Photoshop 6 (Adobe Systems 2001). Interactive two-dimensional animation was developed using Macromedia Flash 5 (Macromedia 2000) and the Web site architecture constructed in Macromedia Dreamweaver 4 (Macromedia 2000). Web-based evaluations were developed using POST-method forms, and collected in a backend SQL database (Microsoft 2000) for later analysis.
In two evaluations media, were scaled for visual complexity. This involved creating renderings from 3D models that were either relatively photorealistic in their approach or more schematic. The photorealistic imagery contained real-world textural and lighting cues such as specular highlights, soft shadows, and detailed surface textures. The schematic imagery looked more like line-drawn images, with flat colors and surface contours indicated with black lines.
The site artwork, animations, video and interface were assessed in three on-line evaluations:
Stage 2 - Evaluation 1
- Preliminary interface design, static image complexity
- Static image complexity
- Video evaluated against animation
Stage 2 - Evaluation 2
- Clinical staging module: layout and functionality
Stage 2 - Evaluation 3
- Complexity scaled 3D rotational model· Review of beta site
Figure 1A.
Figure 1B.
Figure 1C. Depiction of anatomical landmarks of the axilla from complex
to schematic.
Figure 2A. Lymphatic mapping with a radioactive tracer.
(Select image above to link to QuickTime movie with audio; turn your sound volume up.)
Figure 2B. Lymphatic mapping with a radioactive tracer.
(Select image above to link to Flash animation.)
Figure 4A.
(Select image above to link to QuickTime VR. Note that file is +1MB.)
Figure 4B. 3-D rotational model of the axilla.
(Select image above to link to QuickTime VR. Note that file is +1MB.)
Figure 5A.
Figure 5B. User interface.
Evaluation 1. Preliminary interface design, static image complexity, video evaluated against animation
Of 54 surgeons contacted by e-mail, 20 (37 percent) participated in the first formal on-line evaluation.
Methods
Participants were first asked to comment on preliminary designs for the user interface of the site. They were then asked to choose the most effective static rendering of the axilla that was scaled to various levels of visual complexity (Figure 1). Participants were also asked to compare a two-dimensional animation depicting lymphatic mapping (a stage in the SNB procedure) with an intra-operative video of the same subject matter. Space was provided in each assessment for entry of freeform comments.
Results
Site Design: Although they were not able to interact with the site, their comments on basic layout, section headings, and type size were informative to the design of the site.
Static visual complexity: Seventy-five percent of respondents
identified Figure 1A (the most photorealistic image) as that which best
communicated the anatomical landmarks bordering the region of axillary
dissection.
Video evaluated against animation: 58 percent preferred the animation,
31 percent preferred the video and animation together and 11 percent preferred
the video alone (Figure 2A). Respondents who were more familiar with the
procedure (those who performed breast surgery >5 times monthly) generally
preferred the animation alone (Figure 2B).
Evaluation 2. Clinical staging module: layout and functionality
Methods
An interactive clinical staging module (Figure 3) was developed to correlate two staging systems for the classification of cancer: the Union Internationale Contre le Cancer (UICC) "Tumor Node Metastasis" classification system and the American Joint Committee on Cancer (AJCC) staging system. The module is comprised of three components: a "tabbed" navigational chart for toggling between the two staging systems, a visual reference that correlates the cancer stage with a depiction of tumor size and nodal involvement, and notes reviewing the relationship of staging to patient eligibility for sentinel node biopsy. The module was tested for functionality: users were given tasks to complete that would test their understanding of the navigation and content. Respondents were also asked to comment on their preference for the arrangement of the three components. In each of the variants the position of the visual reference was changed to determine whether viewers preferred to view the image on the left, center or right-hand side of the screen relative to the positioning of interactive controls and text.
Results
Fifty-eight percent of respondents preferred Variant 3 where controls
were located on the left, text center and visual media on the right.
Evaluation 3. Complexity scaled 3D rotational model and review of the beta site.
Methods
Eleven respondents (20 percent of the 54 physicians who agreed to participate in on-line evaluation) were asked to compare two navigable QuickTime VR object movies of axillary anatomy (Figure 4). Participants were asked which of the 3D models best demonstrated the three-dimensional nature of the axilla and the relationship of lymph nodes to other structures within the axilla.
Participants were also asked to complete a review of the beta site (Figure 5A & B). We were concerned with receiving feedback on the following aspects of the overall site design:
- Site navigational design, which was meant to facilitate rapid navigation of the site hierarchy.
- The introductory page, which illustrates a case study presented in a shorthand format inspired by physician's case history notations.
- Accessibility of the media files and appropriateness of the media design.
- Maximizing CPD credit allocation while minimizing time requirements (ordinarily, CPD programs grant credits based upon the amount of time that would be estimated to complete the educational activity).
- Additional opportunities for earning CPD credit.
Results
Dynamic visual complexity: Four participants were not able to load the models (either because they did not have the appropriate QuickTime plug-in or because the download time was too great). Of the remaining seven respondents, four preferred model A and three preferred model B. Those who preferred model A sited "greater realism" and "clearer definition" as reasons for their choice.
Beta Web site evaluation:
Respondents were asked to review the beta site and complete a final on-line questionnaire that elicited open-ended responses. A rich resource of constructive feedback resulted. Particular concerns included: legibility of text in some situations; navigational issues; and the reliance on QuickTime for video display. Further modifications to the site were made to address these concerns, including the addition of Windows Media format video.
Development Stage 3: Summative evaluation of final candidate site by expert reviewers
Methods
Accreditation of the SNB course necessitated a thorough review of the final candidate site by four content/target audience experts (1general practitioner, one surgical oncologist, one surgical resident and one general surgeon) who exhaustively reviewed the site for content, pedagogy, and creditability for the Faculty of Medicine, Office of Continuing Education. Concurrently the site was reviewed by the RCPSC.
Results
The RCPSC and the Office of Continuing Education made further recommendations, including the following:
- include clear assertion of specific educational objectives;
- integrate on-line dialogue between participant and the course director (asynchronous learning);
- track participant's time on the site (time participated); and
- include links to full-text downloadable versions of referenced research papers in the topic area for additional CPD credit.
The site was expanded to include these features, and in September 2002 received Section 1 accreditation. The SNB course is one of the first on-line courses to receive "Section 1" approval by an accredited provider of the Maintenance of Certification program of the RCPSC and from The University of Toronto's Faculty of Medicine's Office of Continuing Education. Section 1 courses must include accredited group learning activities and as such are more difficult to obtain since they usually involve costly travel to a conference or enrollment in distance education programs.
Discussion
Our experiences with the process of needs assessment and evaluation have raised many issues with regard to response rate. Although a response rate of 24 percent with the initial needs assessment was considered quite promising, the response rate declined to 3 percent in the final evaluation. There may be many factors that contributed to the declining response rate. The needs assessment was conducted by more traditional methods (a survey was mailed out and returned in a postage paid envelope). During the course of evaluation participants were contacted by e-mail. It may be that respondents were more receptive to a tangible, paper-based survey. Additionally, the evaluation of the site was undertaken over the course of one year. Perhaps this was too long a timeframe, since during this time our response rate dropped by 50 percent. Although the participatory design model necessitates ongoing evaluation throughout project development, measures need to be integrated to ensure continued participation. As on-line evaluation becomes more the accepted norm, the development of clear protocols for successfully collecting data would be of great benefit to media developers.
The response rate to our evaluation of visual media, and the specificity of the subject matter and audience, makes generalizing the results impossible. However our evaluations do raise interesting questions that may further guide the direction of our research efforts. The finding that participant preference for more visually complex models of the axilla came as somewhat of a surprise. We had hypothesized that users would prefer a visual representation of the axilla where extraneous detail was stripped away and structures were more clearly delineated. While this tells us that participants preferred a more "realistic" rendition of the subject matter, we have yet to determine whether these subjective preferences correlate with more objective measurements of media efficacy. This is certainly an area that merits further investigation.
Although the results of the evaluation of animation compared with video was less surprising (results showed a clear preference for animation, though many participants requested that video still be included), it is interesting to note that those participants who performed breast surgery with the greatest frequency requested animation without video. Perhaps these participants are at a level of familiarity with this procedure where video documentation of the procedure is considered less necessary. It may be that this finding merits investigation into the resources that these participants have at their disposal. Increased accessibility of consumer level digital video camcorders have made video documentation of surgical procedures a more commonplace feature in today's operating theater.
Participant preference for a Clinical Staging Module layout of 'Controls > Text > Image' is in keeping with conventions of both Web-based and print-based media. In Web-based media controls are often found on the left, and in print-based media layout is often governed by the left-to-right reading order of western script systems.
Conclusion
The introduction of courses for CPD credit on the Internet will promote state-of-the-art training for surgeons and other health-care professionals internationally. The Internet has created a venue for providing relevant, "just in time" learning. In the development of this CPD course we've found the participatory design process to be invaluable to that end. Needs assessment is critical in directing the production of up-to-date, applicable and practical course material. Ongoing contact with the target audience is essential in the development of successful educational tools, although continued participation (possibly through more rapid prototyping) needs to be ensured.
While a more user-centered design process is well established the evaluation of interactive media, it has not been largely applied to the design and evaluation of visual media. As the Internet continues to grow as a venue for health information and education, so too will the demand for media-rich content. Further research in this area may greatly impact the design of visual media in the health domain.
Visit the Sentinel Node Biopsy Continuing Professional Development Course on the Web at http://www.bmc.med.utoronto.ca/cpd/GEN/SNB/index2.html
References
Adobe Systems. Adobe After Effects Ver. 5.5. 2001 San Jose, C.A.: Adobe Systems Inc.
Adobe Systems. Adobe Photoshop Ver. 6. 2001. San Jose, C.A.: Adobe Systems Inc.
Cantin, J., Scarth H., Levine M. , Hugi M. 2001. Clinical practice guidelines for the care and treatment of breast cancer: 13. Sentinel lymph node biopsy. Canadian Medical Association Journal.165(2):166-73.
Maxon Computer. Cinema 4D Ver. 7.3 (C4D XL 7.3). 2002. Newbury Park, C.A.: Maxon Computer GmbH.
Discreet. Cleaner Ver. 5 (Cleaner 5). 2001. San Rafael, C.A.: Autodesk, Inc.
Dwyer, F. M. 1972. A guide to improving visualized instruction. University Park, PA: State College, Pennsylvania State University, Learning Services Division.
Hirsch, K., McConathy, D. A. 1986. Picture Preferences of Thoracic Surgeons, Journal of Biocommunication. 13(1): 26-30.
Hooimeijer, B. 2001. An Alternative Cartoon Render Technique. Available on the World Wide Web at: http://www.xs4all.nl/~naam/techniques.html
Hugo J. 2000. A grading model for media appropriateness and cultural sensitivity in health education. Journal of Audiovisual Media in Medicine. 23 (3); 103-109.
Macromedia. Dreamweaver Ver. 4. 2000. San Francisco, C.A.: Macromedia Inc.
Macromedia. Flash Ver. 5. 2000. San Francisco, C.A.: Macromedia Inc.
Maibach E, Parrott RL. 1995. Designing Health Messages. London: Sage.
McCready D.R., Bodurtha A.J., Davis N.L., Meterissian S., Robert J., Temple W.J. 1999. Sentinel lymph-node biopsy in breast cancer. Canadian Journal of Surgery 42:406-7
Nielsen J, and Landauer T. 1993. A mathematical model of the finding of usability problems. Proceedings of ACM INTERCHI'93 Conference, pp. 206-213
Peeck, J. 1987. The role of illustrations in processing and remembering illustrated text. In The Psychology of Illustration. Edited by Willows, D. M., Houghton, H. A. New York: Springer-Verlag.
Preece J (ed). A Guide to Usability: Human Factors in Computing. 1993;13-14
Schwalm, R. 1970. Visual communication education. In Visual Literacy, ed. C.M. Williams, J.L. Debes. New York: Pitman Publishing Corporation.
Microsoft. SQL 2000. 2000. Redmond, W.A.: Microsoft Corp.
Tones K, Tilford S, Robinson Y. 1993. Health Education: Effectiveness, Efficiency and Equity. London. Chapman & Hall.
Acknowledgements
This research is supported by a grant from Bell University Laboratories, a collaborative research program funded by Bell Canada, the Ontario Research and Development Challenge Fund and The University of Toronto.
We are indebted to Juho Park and Lawrence Spero of the Bell University Laboratories Health Communications Lab for their assistance in the development of our on-line evaluations. We are also most grateful the Health Communications Lab for funding Meaghan Brierley's work on this project. The authors would also like to acknowledge Bruce Youngson of the Dept. of Pathology at The University Health Network for his generous contribution to the course material.
SNB Course Director:
Dr. Dave McCready, Associate Professor of Surgery, University of Toronto and Princess Margaret Hospital, The Toronto General Hospital (University Health Network) and Women's College Hospital.
The following surgeons/physicians contributed and reviewed the on-line course:
Ruth Heisey, Stephen Tulk, Sean Yong, Barry Anderson and Barbara LeBlanc who reviewed the course on behalf of the University of Toronto's Faculty of Medicine's Office of Continuing Education.
Craig Campbell (Director Professional Development, Royal College of Physicians and Surgeons of Canada)
Douglas Wooster (Chair of the Royal College of Physicians and Surgeons of Canada CPD Standards Committee)
IVIS Group was established two years ago within the Division of Biomedical Communications (BMC), Department of Surgery, Faculty of Medicine, University of Toronto. Our mandate is to develop and research visual media within the health domain. The IVIS Group are:
Jodie Jenkinson, Assistant Professor
Nicholas Woolridge, Associate Professor
Linda Wilson-Pauwels, Professor and Director of BMC
Meaghan Brierley, Research Associate
Co-authors
Division of Biomedical Communications, Department of Surgery Faculty of Medicine, University of Toronto
Jodie Jenkinson, B.A., M.Sc.BMC, is an Assistant Professor specializing in instructional design in BMC. She is the author and designer of The Prostate Centre, a CD-ROM and Web site and co-author of the BreastMatters Web site, both educational programs designed for patients of The University Health Network. She is a co-founder of the IVIS Group.
Nicholas Woolridge, B.F.A., B.Sc.AAM, M.Sc., is a tenured Associate Professor specializing in animation and multimedia in BMC. He is the co-author of Anatomy 300/303 Interactive Lab Companion, a hybrid Web/CD-ROM program. Woolridge has a cross-appointment to Sheridan College's School of Animation, Arts & Design. He is a co-founder of the IVIS Group.
Linda Wilson-Pauwels, A.O.C.A., B.Sc.AAM, M.Ed., Ed.D., is Chair of and Director of the Master of Science in Biomedical Communications (MScBMC) program. Wilson-Pauwels is a tenured, full Professor in BMC with a cross-appointment to Sheridan College's School of Animation, Arts & Design. She is a visual communicator with advanced degrees in higher education. Wilson-Pauwels is co-author of Cranial Nerves and Autonomic Nerves, B.C. Decker Publisher. She is a co-founder of the IVIS Group.
Meaghan Brierley, B.F.A., P.G.Dip. Medical Photography, M.Sc.BMC, is a research associate with the IVIS Group and an instructional media designer and developer with the BUL Health Communication Research Laboratory at The University of Toronto.
Division of General Surgery, Department of Surgery Faculty of Medicine, University of Toronto
Dave McCready, MD, is an Associate Professor of Surgery on staff at the Princess Margaret Hospital (PMH), The Toronto General Hospital (University Health Network) and Mound Sinai Hospital. His specialty interests include breast cancer and melanoma. He holds the Gattuso Chair in Breast Surgical Oncology and is Head of the Breast Site Group at Princess Margaret Hospital.
Correspondence: l.wilson.pauwels@utoronto.ca