Learner evaluation of an online continuing medical education course for general practitioners

Societal expectations for quality medical care have made it imperative that doctors undergo Continuing Medical Education (CME) to keep abreast of new developments in medicine. Taking cognizance of the potential of „Online Learning‟, the OUSL offered an online CME course on cardiovascular health for physicians. The course purchased from Monash University, Australia was updated and adapted to suit the Sri Lankan context. The course which was available in print format was converted into an online format on the learning management system „Moodle‟. While designing the online format, the course was enriched by using audio visual material, discussion forums, self assessment activities and an online reflective journal. Online assessments included a variety of assessment methods. The course team functioned as tutor mentors to facilitate learning. All learners (n=21) participated in the evaluation of the course by completing an online questionnaire. The questionnaire had 16 closed questions and seven open ended questions. Responses to the closed questions were gathered on a four point Likert scale. Frequencies were computed for closed questions and content analysis was carried out for open-ended questions. Learners perceived interactive and collaborative learning as the most liked features. They agreed that the audiovisual material enriched the course and that sessions were well structured, easy to understand and interesting. Some disliked essays and reflective writing while others suggested more interaction and audiovisual material to improve the course. Problems faced by learners were mainly of a technical nature. Finally, learners expressed an overall satisfaction with the course and thought that online learning was a rewarding experience. Learner evaluation showed that pedagogy, social environment and technical underpinning should be properly utilised in optimising learning in an online learning environment. The course had provided an enjoyable, satisfying and useful experience for physicians who lack the opportunity of undergoing face to face CME.


Introduction
It is universally accepted that all professionals should undergo a life-long learning process to keep abreast of the latest developments in their respective disciplines.This has become important for the medical professionals due to the requirement for maintenance of standards by licensing bodies and demand by society for quality medical care.While exciting new developments are taking place in the practice of medicine with technological advances in diagnostic tests and new treatment protocols based on clinical trials and systematic reviews, the public is better informed on health matters than ever before due to the immense growth in Information and Communication Technology.Yet, patients need the advice of doctors whom they trust and are knowledgeable to make correct decisions about their own health.In order to fulfill this need of patients, physicians have to undergo Continuing Medical Education (CME) and Continuing Professional Development (CPD).CME and CPD have now become pre-requisites for revalidation of the license to practice medicine in developed countries.In Sri Lanka too, a National Centre for CPD (NCCPD) in medicine was established in 2006.However, the uptake of CME has been consistently low.Surveys have shown that physicians are unable to attend centrally organised CME activities due to constraints of time, distance, loss of practice and income, leave problems, finding suitable locum cover, family commitments etc. (1,2) .Thus it is opportune to develop alternative modes of delivery of CME to enable physicians to undergo lifelong learning.
In the past decade, online learning technologies have gained popularity as a relevant learning method, bringing about the advantage of flexibility in learning, using the virtual space as a facilitator for knowledge transmission (3) .In western developed countries, increasingly large numbers of physicians earned their CME credits by following online CME courses (4,5) .The popularity of online CME is particularly because online learning technologies provide flexible, innovative and effective interventions that help overcome social, scientific and pedagogical challenges that exist to CME uptake by physicians (6) .
Taking cognizance of these facts and the potential of "online learning", the Open University of Sri Lanka (OUSL) embarked on an online CME course on "Cardiovascular Health" for physicians in 2008.This was made possible through funding provided by the Matching Grants scheme of the Distance Education Partnership Programme (DEPP) of the Asian Development Bank (ADB) under the Distance Education Modernisation project (DEMP).

Design and delivery of the course
The "Cardiovascular Health" course was originally from family medicine programme of Monash University, Australia.It was updated and adapted to suit the Sri Lankan context by a team comprising a cardiologist, a lecturer in family medicine and a family medicine professor.The text of the course material was designed and developed to suit the online format by the course team with the assistance of the Educational Technology Division of OUSL.
The course was developed on the MOODLE (Modular Object Oriented Dynamic Learning Environment) Learning Management System (LMS).This system supports the development of constructivists learner centered online learning environments (7) .
The course was updated by incorporating additional text and reading resources and suitable links to authoritative websites.The subject of clinical audit, an important tool in quality assurance was introduced through a PowerPoint presentation with background narration.Participants were expected to carry out a pre-audit and post-audit on management of cardiovascular risk factors to determine whether their performance has improved after the educational intervention following a CME course on cardiovascular health.Pre-audit and post-audit forms were completed online by the learners.Case scenarios were modified to suit the Sri Lankan context.Knowledge acquisition was enhanced through the incorporation of videos of a coronary angiogram and an echocardiogram while blood pressure and ECG recording videos were used for skills development in a virtual learning environment.Discussions and forums were included in most sessions allowing learners to share ideas and build knowledge through learner-learner and teacher-learner interaction.An online reflective journal was incorporated for learners to record the activities that were interspersed within sessions and to write their reflections on the relevance of learning their own clinical experiences.The forum "virtual café" allowed social interaction, and "Help wanted" forum enabled technical problems to be sorted out.Online assessments included MCQs, case studies, essays, an Objective Structured Practical Examination (OSPE), participation in the discussion forums and the reflective journal.A few selected screen shots of the online course are given in figures 1 to 3.  The course had a credit rating of six, equivalent to 150 hours of study time.21 physicians (learners) participated, and 18 completed the course successfully.

Objective
Objective of this study was to investigate learner perceptions on the CME5201 -Cardiovascular Health online course which was the first CME online course to be offered in Sri Lanka.

Method
At the completion of the course, all 21 learners were requested to complete an online evaluation questionnaire which had been designed to identify learner perceptions.The questionnaire had 16 closed questions and seven open-ended questions.Responses to the closed questions were gathered on a four point Likert scale (strongly disagree [1] to strongly agree [4]).The questionnaire addressed learners" responses regarding course content and activities, media elements, support given by tutor mentors, their online learning experience, and their suggestions to improve the course.Frequencies were computed for closed questions and content analysis was carried out for open-ended questions.

Results
Course evaluation revealed many positive findings with regard to online learning of the Cardiovascular health course.Learner responses to closed questions are given in Table 1.

Question
Strongly disagree

Disagree Agree Strongly agree
Active engagement online with tutors and peers --25% 70% Lessons well structured, interesting and easy to understand --40% 50%

Learner perceptions about the course
Most learners (90%) agreed that the learning sessions were well structured, easy to understand and that the content was interesting and interactive.Eighty five percent (85%) of the learners agreed that the videos and other media enriched the course.For a course designed to appear on a weekly basis, learners" response was positive (50% agreeing and 35% strongly agreeing) that the sessions were made available on the given date.Most learners (85%) were of the view that the workload was manageable.
Learner satisfaction of the structure of the course was further revealed through their responses to open ended questions.They highly appreciated the composition and the flow of the course, with special interest being focused on MCQs, videos, case studies, group activities such as forum discussions and summaries that were interspersed throughout the course.With regard to time limits for assessments, majority of learners (85%) stated that the deadlines allocated for continuous assessments were reasonable.However, a drawback specifically pointed out by one learner was that the workload became too much towards the end of each week.

Learner perceptions on learning and facilitation
Most (95%) agreed that the orientation programme and the hands on experience received on how to use the LMS (viz.Moodle) facilitated their interactions in the course meaningfully.
Closed questions revealed strong agreement (90%) that the tutors and mentors including the cardiologist adequately answered their questions by giving useful feedback, further providing evidence on learner support and facilitation.However, while suggesting improvements to the course, one learner stated that the "Q & A" forum should be answered promptly.In addition to the facilitation by tutors and mentors, closed questions also showed 90% agreement that the course administrators helped overcome the technical problems.
As far as peer support was concerned during the learning, 95% of learners agreed that they had actively engaged in the online course by interacting with peers as well as tutors.This was further confirmed through responses to open ended questions which stated the following among the most liked features in learning.Group activities and discussions.
Exchanging ideas in forums.Ability to involve everybody to enrich our knowledge.
Some other learners identified activities such as self assessment questionaires, case studies, journal activities as the most liked features in learning.However, some disliked writing essays and making daily entries in the reflective journal.The learners identified the "Heart failure" and "ECG" sessions as the most liked sections in the course.

Problems encountered and suggestions
Problems encountered in online learning were obvious through responses to open ended questions.These problems were mainly technical, which they had encountered while making journal entries and uploading assignments in particular.Learners listed their problems as follows: "I encountered difficulties in making entries in the online journal for reflecting and activities half way through the course.""Computer skills were Nil when I started this course.Therefore I spent more time" "Internet problem" "Some videos were not accessible" Giving suggestions to improve the course, learners recommended technical improvements to the course design.They suggested to have an alternative to maintenance of a bulky online journal.They also wanted more details to be added to improve the ECG and atrial fibrillation sections.Further, they suggested adding more self assessment questionaires, case studies, more lectures/presentations in video format, and more interactive sessions.

Reflections about online learning and outcomes
Finally, learners were asked to reflect on their experience about online learning.Almost all learners highly valued "online learning", having had the opportunity to learn independently overcoming barriers of time and place and stated that learning had been a rewarding experience.
"Very rewarding experience, convenient method, can learn at one"s own pace and there is opportunity for interaction" "It is a very easy way of doing a course -my time, my place and my pace and I hope you would start another online course like this" "Very good thing to start in Sri Lanka……" The positive learning outcomes of the "cardiovascular health" course were also evident from their reflections.For example, learners stated that they had improved their management of patients with cardiovascular disease (CVD), learnt to take more precautions with patients on multiple drug regimes and learnt to carry out audits on improving the management of cardiovascular risk factors.

Discussion
These findings brought to light a number of factors that have contributed to the success of the online CME course.

Collaborative learning
One of the main factors for the success was the activity based collaborative learning opportunity provided through discussion forums.Majority of the learners identified such collaborative learning activities as the most liked feature.The reason for this may be because discussion forums enabled the learners to remain connected together facilitating "learnerlearner" as well as "learner-teacher" interactions.It had also made learners exchange ideas, argue different viewpoints, receive feedback, formulate solutions to given problems and get specific clarifications from the consultant cardiologist who was the expert and a mentor in this course.This indicates that forum discussions had allowed social construction of knowledge within a learning community.As shown by Lopez-Islas (2004) (8) , such collaborative learning experiences facilitate construction of knowledge through sharing of ideas within the group.Studies in other disciplines too have found that the use of activity based learning and shared understanding with peers within collaborative online communities offer promising potential for professional development ( 9,10) .

Self learning activities
In addition to the collaborative learning opportunity provided through forums, another feature that was preferred by the learners were the self learning activities (viz.online questionaires and reflective journal) provided within the course.Questionaires with prompt feedback engaged learners in thinking, problem solving, responding to questions and making judgments of their own learning.Request for more quizzes show that they were a motivating factor facilitating learner autonomy.The fact that some learners liked making journal entries is encouraging, as it is a feature that facilitates knowledge construction.While reflecting, learners are able to identify the rationale of their own thoughts, relate new knowledge to prior knowledge, find out their weaknesses and strengths and finally formulate their thoughts within a context that is related to their own life and experiences.However, a few learners disliked making journal entries and writing essays which could be related to time constraints.

Learning with media
Inclusion of media components such as audio visual (AV) material and other reading material were also regarded as important and liked features in the course.Use of video technologies in this course provided the visual and audio realism of a face-to-face class, facilitating skills development in blood pressure measurements and recording of an ECG.The videos of a coronary angiogram and echocardiogram assisted learners" in acquisition of knowledge within a limited period of time, which would not have been possible through the text mode in distance learning.Aluwihare"s work in Sri Lanka, also reported the successful use of video material for skills development in a distance mode CPD course for surgeons (11) .
The use of background narration in videos/PowerPoint presentations was another feature that may have provided better learning environments than visuals alone, as this would have facilitated the social presence of teacher immediacy in an online environment (12) .

Problems and suggestions
As far as the problems encountered by the learners are concerned, their main difficulty was making entries in the reflective journal.This was because learners had to enter all their learning reflections, all activities and continuous assessments in one "global" journal which eventually created a bulky online journal.Taking cognizance of this difficulty, suitable adjustments were made in the design of the reflective journal for the next batch of students.Other technical problems encountered by the learners such as internet access and downloading of videos were sorted out through the help wanted / help given forums by course administrators.
Some learners stated that they were novices in a technology mediated learning environment and they had to spend time in familiarising themselves to this mode of lesson delivery.This has been identified as a discouraging factor for physicians to enroll in online CME courses and a common obstacle faced by CME providers who initiate online programmes (5) .However, by following this online course, these learners may have benefited in developing their computer / media literacy skills which is becoming an important trend in the 21 st century.In future, these difficulties could be minimised by providing the necessary knowhow about the technical requirements and in providing hands on skills necessary for online learning at the beginning of the course.
Learners" suggestions also pointed out some essential motivating factors for learning.Request for more narrated lecture presentations and videos as well as self assessment questionaires and interactive sessions show that learners need to read, view, listen and also actively engage in doing something within the learning environment.This indicates that online environments optimise learning for those with different learning styles, visual, auditory or otherwise.

Learner satisfaction and results
Finally, in their reflections, learners indicated that this online course had been informative, interesting and motivating.Learners were genuine in their appreciation of the course, having perceived it as a rewarding experience that permitted learning while attending to their work and domestic responsibilities.Curran & Fleet (2005) (5) have also identified this kind of learner satisfaction in other CME online courses.Learners" reflections also revealed that the course had addressed learners" continuing educational needs.It seemed to have had a beneficial impact on clinical practice, particularly in diagnosis and management of patients with cardiovascular disease.Therefore, as pointed out by Keller & Suzuki (2004) (13) , this course too had provided knowledge related to learners" needs within a personalised learning milieu to master the learning tasks, and the skills to apply what they had learnt in actual practice.

Conclusion
The course had provided an enjoyable, satisfying and useful experience for physicians who lack the opportunity to experience face to face CME.Lessons learnt from this evaluation have been used to improve subsequent online CME courses.This study also shows that pedagogy, social environment and technical underpinning should be properly utilised in optimising learning in an online learning environment.
This course offered for the first time by the OUSL, was an eye opener to the tremendous potential of "online learning" for general medical practitioners.We hope that this is just the beginning of future developments in continuing medical education for physicians in Sri Lanka, perhaps in collaboration with medical professional organisations, the Ministry of Health and the Sri Lanka Medical Council.

Figure 1 .
Figure 1.Screen shot of the "Home page" of the course

Figure 2 .Figure 3 .
Figure 2. Screen shot of the forum "Discussion Q & A"

Table 1 .
Learner responses to the closed questions (n = 21)