An Evaluation of 
the Commonwealth of Learning section 
on Elearning for International Organisations

by Dr Rory McGreal PhD
July 2008

Table of Contents
							
1. Executive Summary 							i

2. Project Context								1

3. Monitoring & Evaluation Purposes					4

4. Monitoring & Evaluation Design & Implementation			4

5. Findings									6

6. Conclusions								18

7. Recommendations							20

8. Annexes									21

Annex A Learning history								22
Annex B Survey Monkey: Learner survey						34
Annex C Consent agreement								36
Annex D Interview questions								37
Annex E Terms of reference								40
Annex F Contributor anonymity statement					42
Annex G Reports and publications consulted					43
			
1. Executive Summary

Project description
This evaluation of the Commonwealth of Learning’s  (COL) eLearning with International Organisations (eLIO) section includes a backgrounder, a statement of the purposes of the evaluation, the monitoring and evaluation purposes, the findings,  the plan’s limitations, the audience for whom it is intended, the results that have been tracked and  a discussion of the implications, followed by conclusions and recommendations. It is intended primarily for use by the COL eLIO manager, the COL senior management, the COL Board for analysis and planning and the partner organizations. 
Monitoring and evaluation purposes
The purpose of this evaluation as expressed in the work plan is “to carry out an evaluation of the eLearning for International Organizations Initiative, including the development and implementation of an evaluation plan and synthesis of an evaluation report.”

Intended Outputs/Outcomes for the Activity
The eLIO have achieved their results, yielding significant benefits, developing a distance learning model for potential application in a variety of different (but possible) organizational and educational settings or environments with some important additions to the stock of open learning pedagogical knowledge. To assess the benefits of the first iterations of the eILO programmes, the section engaged evaluators to examine the programmes and report on the benefits and problems encountered, making recommendations.  These evaluations can be seen as case studies of each programme. The projects were similar and the evaluations tended to cluster around the same set of benefits and recommendations. For example stressing the high quality of the instructional materials and the importance of the tutor – learner relationship. 

Findings
 For the purposes of this study, the evaluator has attempted to specify in his analyses the role that the eLIO section has played.  Based on this assessment a conservative judgment has been used to characterize the section’s contribution for purposes of developing findings and recommendations.  What cannot be dismissed in any analysis of the work of this section is the fact that all the stated results were met and in some cases exceeded, coupled with a rather overwhelming degree of satisfaction expressed by all the participants, including the learners, the partners, the tutors, the COL staff and the eLIO section employees.
Limitations
This plan is limited to an evaluation of the activities of the COL eLIO . The external validity or generalisability is extremely limited.  Reliability is always an estimate and is dependent on the testimony of the people interviewed,. However other sources have been consulted (evaluations, reports, plans etc.)  which tend to confirm the oral histories.
Conclusions
The results show that the eLIO section has succeeded in achieving its goals, including
•	Teaching more than 2000 learners who showed a high degree of satisfaction;
•	Partnering with eight international organizations, who were very satisfied with the programmes;
•	Achieving a 62% female participation rate in programmes;
•	Sustaining a high completion rate of 75% average across the board
•	Two new elearning courses have been developed and customised;
•	Requested needs analyses of the participating organizations were conducted;
•	Training materials have been tested and piloted;
•	Highly qualified tutors have been recruited, trained and supported;
•	Programme evaluations have been ongoing and intensive;
•	Simple appropriate technologies have been harnessed for course delivery;
Shortcomings of the programmes, although minor in comparison with the overall success of the eLIO section’s initiatives include the following:
•	No real pre- and post-tests to measure learners’ achievement;
•	Pricing structures for the programmes were not thoroughly analysed in order to  determine if the pricing was reasonable in the market;
•	The instructions in the materials are not as clear and direct, nor do they always model good practice in the use of plain English;
•	Limited attempt was made to secure copyright licensing rights; and
•	Little use of OER software falling short of its goal.
Recommendations 
Taking into account the general level of satisfaction and increasing number of clients, the following recommendations are in order:
1.	Keep on doing what you are doing.
2.	Edit and revise the material to model plain English
3.	Re-examine your pricing structure to maximize revenues as much as possible.
4.	Negotiate with partners either copyright control or an open license.
5.	Implement new technologies to support the learning management and administration (LMS, CRM).
6.	Develop a succession plan to cover for the section if the present manager leaves.

2. Project Context 
Outcomes and outputs and related success indicators
The original outcomes and outputs were as stated below:
Outcomes
Development and delivery of technology mediated distance learning programmes with international organisations to enable access to learning by a dispersed and diverse international workforce.
o	By 2009, at least 6 major international organisations active in delivering the MDGs have formed a partnership with COL in customising gender and culture sensitive workplace eLearning programmes.  
o	An additional 2,000 adult learners (a 6 to 4 ratio of women and men) will be trained by 2009.
o	At least two new contextually relevant sets of ODL/ICT-based learning materials for occupational skills development created and each is taken up by at least two major institutions.
Did the COL eLIO achieve its goals through the activities posited? Activities:
o	Actions to achieve the outputs
a.	Analyse and understand the learners’ needs, learning objectives and outcomes, the learning environment and technology infrastructure of the organizations.
b.	Design courses and develop materials through customization.
c.	Test all training materials and delivery designs
i.	Build responsive infrastructure and clear processes for course production and presentation. This includes: 
		       * preparing storyboards/templates;
* using robust, affordable and user-friendly technology and     
   media;
	* recruiting and maintaining a cadre of trained and        				   committed tutors;
	* using OERs software as far as possible.
Monitoring Data
a.	pilot evaluation reports
b.	monthly progress reports
c.	tutor listservs
d.	end of course evaluation feedback
e.	programme evaluation studies
Does the existing data gathering mechanism adequately cover the necessary information for analysis and reporting? Where are the gaps? 
The time allotment by COL eLIO staff was set at 2% for initiative monitoring and evaluation. The staff budget was capped at 25% of the revenue generated. Was this goal achieved? By how much?
Outputs						Success Indicators

•	number of participants in programmes		+2000 by 2009
•	number of organizations served			6
•	number of regions served			not stated
•	  revenue							self sustaining by 2009

Activities implemented
The eLIO team developed and delivered technology mediated distance learning programmes with international organizations in different regions around the world. They formed partnerships with international organizations in order to accomplish this. The learning materials were developed and customized for each organization to promote the learning of relevant occupational skills, based on an analysis of the learners’ needs and the capacity of the technological infrastructure available. Qualified tutors were engaged and trained in order to facilitate the delivery of the programmes.

Partners involved
The following organizations participated as partners with eLIO:
1.	World Health Organisation
2.	UN High Commissioner for Refugees
3.	International Labour Organisation
4.	International Federation of the Red Cross and Red Crescent Societies
5.	The Joint UN co-sponsorships for HIV/AIDS 
6.	World Bank
7.	UNICEF, India Country Office
8.	Office of Internal Oversight Services (OIOS)

Acknowledgements
This evaluation of the Commonwealth of Learning’s  (COL) eLearning with International Organisations (eLIO) section was conducted by Rory McGreal, Associate Vice President Research at Athabasca University.  It was commissioned by Angela Kwan, the Learning Manager of eLIO, who worked with the evaluator assisting in the implementation, analysis, and reporting aspects of the evaluation.

Limitations 
This plan is limited to an evaluation of the activities of the COL eLIO and does not address the wider issues associated with the COL organization. The evaluation is for this particular section only and it is recognized that the external validity or generalisability is extremely limited.  Reliability is always an estimate. It has been addressed by triangulating the data collected using at least three different programme case studies, documents, a survey and interviews with a random sample of participants, COL eLIO staff, and others using similar questions and interviewing techniques. 
The validity of the evaluation conclusions, in this case cannot be assured by a control group.  There is often a multi-group threat to the validity of an evaluation, when comparing different groups taking the same programme. As this evaluation did not involve comparing the outcomes of the different groups, but rather an overall consideration of the section’s activities, including the programmes with different organization, this was not a problem.
3. Monitoring Evaluation Purposes
This evaluation of the eLIO has been commissioned to provide an external evaluation of the  development and implementation of the eLIO programmes, reach conclusions and make recommendations. The audiences for this evaluation are the COL eLIO team leader, the COL senior management, the COL Board for analysis and planning and the partner organizations. A secondary audience would be the COL staff, other COL service users, and the wider distance education community.
4. Monitoring and Evaluation Design and Implementation
The evaluation was conducted using triangulated data collected from several different pilot and programme evaluations, published papers, a simple survey instrument, and other documents as well as interviews with a random sample of participants (student, partners, tutors). The COL staff, and a sampling of others were interviewed using similar questions and interviewing techniques.  These included seven COL staff member,  five tutors, three partners and 10 learners. The interviews were collated into a “Learning History” document (Annex A). The learner survey is available in Annex B.
The information collected consisted of documents and previous project evaluations supplied by the COL eLIO, the COL and other relevant web sites, published articles, and from interviews with the COL eLIO staff, other COL personnel and a small sample of participants in the COL eLIO projects.  Documents were obtained directly from the eLIO and from Internet searches. 
A random sample of representatives from three  of the eight participating organizations were contacted and interviewed. A random sample of tutors (five from a list of twenty) were also contacted and interviewed. Thirty students were randomly selected and asked to participate in an online survey using Survey Monkey. Ten responded and one agreed to be interviewed by telephone.  The random samples were elicited using the application available at Random.com. The participants in each group were given a number and then the numbers were input into the random.com application, which randomized the total and picked a sample. The participants contacted were sent a consent agreement (Annex C).
A list of questions (Annex D) was used as the main instrument for the interviews. Other supplementary questions were used as they emerged out of the discussion with participants. The analysis included an examination of the question responses using the 6R assessment criteria (Relevance, Results, Reach, Regions, Resources and Relationships). These were triangulated with the documentation and survey.

The interviews were conducted either by telephone or in a face-to-face format in June 2008. The information was collected, converted into a Learning History format and analysed for this report.  A learning history methodology provides organizations with a special approach to learning from the experiences of the participants. In the right hand column, relevant observations are described by the people who took part in them. The participants are quoted directly, and their statements are placed in a category defined by their role to increase anonymity. The researcher then attempts to weave the words into a coherent story based on the information provided, including facts, results and assumptions of the participants. This methodology is based on approaches and techniques in a variety of fields including organizational learning. 
This evaluation consists of a summative assessment of the outputs and achievements of the COL eLIO; for formative input for planning future programme implementations; as well as expanding partnerships and increasing the efficiency and effectiveness of the sector. The key performance indicators – output and outcomes have been identified from COL documents and in consultation with the COL eLIO representative.  These include information on the organizations served and the participants in the programmes delivered.  Appropriate documentation and access to interviewees was made available to the evaluator who developed a strategy for analysing the data provided.  Indicators included data on the growth of the initiative (e. g. number of courses created and delivered, number of learners, learner achievement, engagement of technology, lessons learned, threats & opportunities, expansion, gender equity, etc.).
This evaluation can be used by COL to inform their planning processes and to report on activities to the COL stakeholders. As requested, this report makes use of the 6R filter to assess the quality, outcome and output of the initiative.

5. Findings
Relevance
There was a consensus among the COL staff, perceiving the eLIO to be marginalized or “peripheral”, primarily because it is the only section of COL that has been mandated to be self-supporting. They note that the eLIO, in fact, contributes significantly to the core mandate of COL as expressed in the COL Three Year Plan (Commonwealth of Learning, 2006). The eLIO’s relevance to COL is something more than revenue generation.There is a strong alignment of its goals in serving international organizations and those of COL in increasing capacity and demonstrating the effectiveness of  open and distance learning (ODL). It serves as a successful example of the use of technology for ODL in developing countries including the Commonwealth. The section is responsible for substantially adding to the reputation of COL as a professional organization serving the developing world using the best approaches and techniques in DE. As an organisation that works with governments at the policy level for human resource development, COL’s reputation is enhanced when the eLIO  at the grassroots level, demonstrates the viability, effectiveness and sustainability of international ODL.  COL “needs something to crow about”.
Unexpected outcomes
The eLIO has also demonstrated its relevance by its actions, when other opportunities come to COL because of its reputation for delivering ODL. There is a wider opportunity for COL in HR development from UNICEF (possibly $1.9 million) presently being discussed along with a major capacity building project for South African civil servants.  These opportunities have come to COL by word-of-mouth. Without advertising, the eLIO programmes have built viable relationships and increased COL’s visibility among international organisatons, particularly major funders.
Results
The results to be measured include in the first instance the stated intended results of the eLIO section as per the COL strategic plan. They are listed below along with information on the success of each.
Outcomes
o	Development and delivery of technology mediated distance learning programmes with international organisations to enable access to learning by a dispersed and diverse international workforce.
Successful: The eLIO developed and delivered several distance learning programmes to international organizations with widely dispersed and diverse international workforces.
o	By 2009, at least 6 major international organisations active in delivering the MDGs have formed a partnership with COL in customising gender and culture sensitive workplace eLearning programmes.  
Successful: The eLIO surpassed this goal, partnering with eight international organizations active in delivering elearning supporting Millennium Development Goals. These have been generally described as being sensitive to gender and cultural issues in the workplace. The following organizations participated as partners with eLIO:
1.	World Health Organisation
2.	UN High Commissioner for Refugees
3.	International Labour Organisation
4.	International Federation of the Red Cross and Red Crescent Societies
5.	The Joint UN co-sponsorships for HIV/AIDS 
6.	World Bank
7.	UNICEF, India Country Office
8.	Office of Internal Oversight Services (OIOS)
o	An additional 2,000 adult learners (a 6 to 4 ratio of women and men) will be trained by 2009.
Successful: 1766 adult learners (62% female) have received training from the eLIO section. This is on track to exceed 2000 learners by 2009. 
o	At least two new contextually relevant sets of ODL/ICT-based learning materials for occupational skills development created and each is taken up by at least two major institutions.
Successful: Two new elearning courses have been developed adapting the materials and contextualizing it to different organizations. These are the following:
Did the COL eLIO achieve its goals through the activities posited? Activities:
o	Actions to achieve the outputs
a. Analyse and understand the learners’ needs, learning objectives and outcomes, the learning environment and technology infrastructure of the organizations.
Successful: Needs analyses were conducted with the partner organizations, taking into account the learning environment and technological infrastructure. Adjustments to the courses were made based on these analyses, for example changing deadlines, examination schedules, use of CDRoms for distributions and email rather than more complex applications on the Internet.
b. Design courses and develop materials through customization.
Successful:  Six courses were designed and adapted specifically for each different organization, taking into account their different cultures and learning environments. All partners interviewed agreed that this was done very well and according to their needs.
c. Test all training materials and delivery designs
i.  build responsive infrastructure and clear processes for course production and presentation. This includes: 
			ii.  preparing storyboards/templates;
Successful: One partner reported on the difference between her organizations approach to instructional design and that of eLIO, specifically referring to eLIOs change in approach to accommodate their wishes. This included adopting storyboards and templates as part of the learning design process.
iii.  using robust, affordable and user-friendly technology and     
   media;
Successful: The eLIO course designers deliberately focused on low technology solution in order to ensure that the course would be robust, user-friendly, affordable, and most importantly, accessible by the learners dispersed around the world in developing countries.  This included the distribution of CDRoms for the course materials and the use of simple email for communications.
 	Iv,  recruiting and maintaining a cadre of trained and        				   committed tutors;
Successful: The eLIO recruited and maintained a cadre of trained and committed tutors. The partners and students unanimously agreed that the tutors were indispensable and the most important part of the entire programme.
	v.  using OERs software as far as possible
Not Successful: No Open Educational Resource (OER) software was used.  The project was limited by the proprietary software available in the partner organizations. For example MS  Office, including MS Word was a standard that was known and used by the participants. 
d. Monitoring Data
i.	pilot evaluation reports
Successful: A selection of pilot evaluation reports have been made available to this evaluator (Commonwealth of Learning, 2003; Murphy, 2003; Ruhe, 2002; Shale, 2003)
ii.	monthly progress reports
Successful: Monthly progress reports supplied by the tutors in a standard format were a regular occurrence. A random sampling of these reports were provided to the evaluator.
iii.	tutor listservs
Successful: Tutors were able to communicate with their students, the administration and the administrators using listservs. In addition a web conferencing application was also made available for discussion groups.
iv.	end of course evaluation feedback
Partially successful: Learners were asked in the evaluations to provide their views on the course and their estimation of its success and their attitude towards it.  They were satisfied with their progress and very positive in their attitude towards the course and the tutors in particular. However, other than some very tenuous pre and post-testing, there has been no reliable testing of the achievement level of the learners.  There are problems associated with administering such pre- and post-tests in a distributed environment. Nevertheless, as was pointed out by some of the participants, such a practical test should be an integral part of the programmes in order to show learners, partners, tutors, administrators and outsiders that real progress in achievement levels has been made through the course.
v.	programme evaluation studies
Successful: Programme evaluation studies have been conducted and acted upon (Dunlop, 2005; John, 2005;Walker & Dzakiria, 2005; Wilson, 2006 & 2007).

Does the existing data gathering mechanism adequately cover the necessary information for analysis and reporting? Where are the gaps? 
The evaluations of both the pilots and the full courses were conducted in a professional manner. The learner feedback although difficult to arrange was adequate for analysis with a return rate of 30% to 40%. As previously stated, the only real gap in the learner evaluations is in the need for pre- and post- tests of achievement. 
The time allotment by COL eLIO staff was set at 2% for initiative monitoring and evaluation> The staff budget was capped at 25% of the revenue generated. Was this goal achieved? By how much?
This refers to funding allotted for program monitoring and evaluation and it was achieved.

Output summary
2007 - 2008
o    number of participants in programmes: 1,085
o    number of organizations served: 8
o    number of regions served: 4 
o    Revenue: ~$700 000 Costs:~$500,000; Difference: $200,000 (28%)
		* $578,032 (to Feb. 29,2008)

2006 - 2007 fiscal year:

o    number of participants in programmes: 681
o    number of organizations served: 6
o    number of regions served: 4 
o    Revenue: $544,193 Costs: ~$450,000; Difference: $100,000 (18%)

o	 138 countries; 28 Commonwealth countries
o	15 of the least developed countries

Besides these outcomes and outputs that were goals of the eLIO as set out in the COL Three Year Plan, there were other results that need to be taken into account in any reasonable evaluation of the section. Foremost is the fact that more than 2000 learners became more confident with their English their writing skills. Many of these learners are either in Commonwealth countries (28) or liaise with officials in Commonwealth countries. The student survey results (See Annex B) show that learners overwhelmingly felt that their writing and analytical skills had improved and that the course content was sufficient, clear and easy to understand and appropriate for their organizations. They were particularly emphatic on the value of the tutorial system, informing on the interactivity with the tutors and the need for their help and guidance.

Unexpected opportunities
Certainly, the unexpected opportunities with UNICEF and South Africa are among the unplanned for results that can be used to further validate the section’s work. The very high esteem for COL as expressed by the partners is also important as their word-of-mouth advertising not only led to more partners within the UN system, but also to the above-named opportunities. When partners come back and renew their contracts while boasting (“Thrilled, we found them!”)  of the success to the others, this is real evidence of the success of the programme. The customized curriculum for each organisation along with the personalized service was highly appreciated. The tutor system was also an important result that may not be immediately apparent. Their empathy, flexibility and personal skills, combined with a very tight organizational structure was key to the success of the programme. And, it can be considered to be a key result of the programme itself, when taken as a whole.

Another significant result for the benefit of the partner organizations, which might also be of service to COL if implemented internally, was the programme’s emphasis on standardizing communication practices around simple plain English writing. Partners felt that this helped them to standardize around core functions among the learners who are operating using disparate cultural norms. More widely, as the courses spread to the different international agencies it has become “a first”, one of the rare examples of sharing resources among the different UN organizations.

On the negative side, from the learners, partners and the tutors, while valuing the content highly, they feel there is room for improvement in the instructional part of the content, for example having clear section markers, a lack of variety in the examples and the need for more contextualization. Tutors felt that the materials themselves need to be edited to reflect good practice in plain English.

Reach
Partners are convinced that online courses are the only viable way that they can reach their distributed staff for training. They particularly emphasise this need for ODL in developing countries and for staff who are traveling or who have fragmented work assignments. In this regard, they support the distribution of the courses on CD because of the erratic nature of Internet connectivity in many countries. 

Tutors see a need for organizations to be more pro-active in supporting ODL as a means of extending their training reach. They need to develop a culture of using DE. Tutors could be used to train those within the organisation in ODL skills. In addition, more courses on a wide range of training subjects could be developed and delivered.
Internally to COL, staff members refer to the need for other COL sections to benefit from the expertise and skills of those in the eLIO section. They feel that the section should be reaching out to other COL sections that could use their expertise in capacity building and human resource development. Reaching out can also be achieved successfully by making the most effective use of new technologies as they emerge. The section should experiment more with novel applications as access to them becomes more common among the partners.

Regions
Courses were delivered in four regions of the Commonwealth (Africa, South Asia, the Pacific and the Caribbean). As previously mentioned, among the most significant developments as a direct result of the activities of the eLIO, is the opportunity for a contract for delivering training among the civil servants in South Africa. This could strengthen COL’s core mandate for capacity building, by using the Canadian tutors to train South Africans as ODL tutors.
The different regions could also be strengthened using the technology to create regional groupings of learners so that they can engage in peer-to-peer learning and teaching.

Resources
The ambiguous position of the eLIO has been referred to previously. COL’s emphasis on Extra Budgetary Financing has resulted in the section being considered different from other COL sections. The eLIO may not be considered as “core” by the COL board, particularly as it serves many non-Commonwealth countries. The eLIO is perhaps both core and extraneous and as such is more useful to COL that it might be otherwise. While within the organisation, it provides an opening, a window to other international organizations involved in complementary activities.  Therefore eLIO can be seen as a resource.

The eLIOs emphasis on using simple technologies, while important for promoting access and supporting students, may not be the best strategy in the management and administration of their programmes. They use applications like Survey Monkey for their surveys, so they are taking some advantage of the technologies available. The recent adoption of Basecamp for computer conferencing is another example.  However, there are other open source applications that they might experiment with to improve their management and efficiency of the learning process.  This experimentation with technologies could form part of an extended mandate for the section in the research and development of DE. For example, a Customer Relationship Management (CRM) application may help to improve learner support and administrative efficiencies. Open source application do exist (e.g. SugarCRM, VTiger; Campware).

The Learning Manager
Many of the respondents made a special point in commenting on the professionalism and competence of the eLIO staff, making specific reference to the Learning Manager.  She has been commended for turning her section from an “ugly duckling” to a “swan”. Others refer to the section as being very “person-based”. Others say that she runs a “tight ship” as an excellent organizer and administrator.  Some wonder if the section would have a future if this particular Learning Manager was no longer running it, even suggesting that if she leaves, COL would have to review the section and perhaps re-organize.

Discussion
From the establishment of the COL eLIO a debate has surrounded its role in promoting the organization, developing partnerships, fund raising and support for ODL. An important development in the debate has been the evolving understanding of the section’s role in supporting COL’s overall  policy direction, and hence, the section’s alignment with the organizations goals.  For example the eLIO’s efforts in delivering training to UN agencies has resulted in opportunities for other COL sections from UNICEF and possibly from the South African government. These opportunities can relate directly to COL’s core goal of capacity building in the Commonwealth.

After the last evaluation, COL priorities moved in the direction of Extra Budgetary Funding for the eILO.  In the past, the role of COL, has swung between support of demonstration projects and limited funding of research . In the present three year plan COL has settled into a preference for intergovernmental relations and policy. This very brief recounting of the shifting forces that have shaped COL in the past, at least serves to signal the twists and turns of both COL’s goals and management philosophy that the eLIO Learning Manager has had to follow. Without an appreciation of these shifts, evaluating the successes and failures of the eLIO programme could have been a very frustrating and puzzling enterprise.

6. Conclusions
The eLIO have achieved their intended results, yielding significant benefits in their principal outcomes and outputs, developing a model for potential application in a variety of different (but possible) organizational and educational settings or environments with some important additions to the stock of open learning pedagogical knowledge. 
On the other hand, the section has not employed a consistent methodology for estimating and evaluating the benefits from its programmes. Their evaluations tend to focus on attitudinal benefits regarding ODL and their individual experiences, rather than taking into account the broader array of benefits (realized and otherwise) flowing from their grant work and other initiatives. Finally, the way in which the section’s programmes were organized and managed made a real difference to the benefits that were produced by the initiatives undertaken. 

To assess the benefits of the first iterations of the eILO programmes, the section engaged evaluators to examine the programmes and report on the benefits and problems encountered, making recommendations.  These evaluations can be seen as case studies of each programme. The projects were similar and the evaluations tended to cluster around the same set of benefits and recommendations. For example stressing the high quality of the instructional materials and the importance of the tutor – learner relationship. There were sufficient programme evaluations to reliably inform the analytical framework and to draw reliable conclusions about the success or failure of the overall program. 
Perhaps the most difficult analytic problem is assigning to the section a proportion of the overall benefit of the programmes in a manner that properly reflects the section’s contribution to it. For example, how much of the partner’s and learners’ input was responsible for the overall success? There is no reliable way to accurately quantify the section’s contribution in most cases, and doing so remains a methodological challenge for the future. Although having a pre- and post-test of learner achievement should go a long way towards increasing the reliability of any evaluation. For the purposes of this study, the evaluator has attempted to specify in his analyses the role that the eLIO section has played.  Based on this assessment a conservative judgment has been used to characterize the section’s contribution for purposes of developing findings and recommendations.  What cannot be dismissed in any analysis of the work of this section is the fact that all the stated goals were met and in some cases exceeded, coupled with a rather overwhelming degree of satisfaction expressed by all the participants, including the learners, the partners, the tutors, the COL staff and the eLIO section employees.

7, Recommendations 
Taking into account the general level of satisfaction and increasing number of clients, the following recommendations are suggested:
1.	Keep on doing what you are doing.
2.	Edit and revise the material to model plain English
3.	Re-examine your pricing structure to maximize revenues as much as possible.
4.	Negotiate with partners either copyright control or an open license.
5.	Implement new technologies to support the learning management and administration (LMS, CRM).
6.	Develop a succession plan to cover for the section if the present manager leaves.

8. Annexes

Annex A Learning history
Annex B Survey Monkey: Learner survey
Annex C Consent agreement
Annex D Interview questions
Annex E Terms of reference
Annex F Contributor anonymity statement
Annex G Reports and publications consulted

Annex A

A Learning History of the COL eLIO

Relevance, Results, Reach, Regions, Resources and Relationships

Relevance

There is a consensus among COL staff that although the eLIO is perceived to be marginalized, it in fact, contributes significantly to the core mandate of COL as expressed in the COL Three Year Plan.

They wish that this unit could be better integrated and less peripheral so that the other sectors can also take advantage of the expertise housed in the eLIO.

COL staff recognize  importance of the eLIO in spreading the COL message internationally and  advertising COL as a leader in elearning for developing countries.

There is support for the eLIO as an action research unit disseminating research finding on how to support elearning online for developing countries.

eLIO work has the potential for gaining other contracts for COL eg. Capacity building by training elearning tutors in South Africa.

The eLIO is expected to be self sustaining and yet is limited in its ability to bid on contracts not wanting to be seen as competition with other Commonwealth agencies.	COL Staff

Internal
Alignment with COL goals is strong;
eLIO is marginalized  because of its emphais on cost recovery; it is more limited as other sectors are working at the international governmental policy level;
But, eLIO is better aligned than we communicate ; a demonstration unit of what we can do.- aligning elearning to customer needs – it’s focused on the user group;
called a “cash cow” IO MUST be positioned as fulfilling a COL need

IO needs more support; it is a stepchild in the COL program – not a useful program
IO is core and central to the mandate of COL to promote learning through technology; it is part of human resource development and so is mainstream It is capacity building
Initiative is directly linked to the 3 year plan; bids must align with 3 yr plan
Bids are internally reviewed by Manager
IO should be core funded but self-financing
First and foremost this is DE showing ped strategies of DE -  very close fit
IO is seen as peripheral no matter what the spin, countries are no asking for it.

Screws turned down to recapture overhead in the “wake of COL International disaster”
Is IO in tune with COL’s current emphasis or is it an outlier
Do more research with own system.

External:
Outreach is part of COL’s mandate
IO is good advertising for COL
Used to have bigger brief to bid against anything NOW it is tailored to particular training with strategic orgs in development
in non-Commonwealth countries. IO must be more accountable to COM donors
they can’t compete with other Commonwealth institutions

“sort of” – IO exemplifies best practice in DE – a demonstration of what is possible ; it gives visibility beyond the commonwealth

If it is seen to be just funder money Board might shut it down
IO should not be a cash cow; it shows off DE; it is experimental and demonstrates possibilities; It shows what COL can do. It should become more of an action research arm showing pragmatic ways of getting learning to learners in developing countries

And be more active in dissemination of research.very important “COL needs something to crow about”  A proof that DE works in intl development

Could eLIO exist without COL?
•	other UN orgs added to first by reputation
 
*COL should be able to demonstrate the sustainability of DE – It can be done this is proof
Some say COL is about policy more strategic
COL needs to be about capacity building as well as policy
Ss apply learnt skills directly in their work.
More requests are coming

UNICEF and South Africa : possible contracts are good for COL ;  addresses COL’s mandate for capacity building

Positive perception of COL by people touching this part of the elephant.
Need capacity building – use Cdn tutors to train others in developing countries
Need to empower the learner

Care and caution needed in marketing – constraints in terms of expansion
eLIO trains people in field offices that deal with COL

Spin it for development  and capacity building all the orgs are in development

A good example of what COL can do more of
Program has grown by word of mouth
Built relationships; increased visibility

IO is “a feather in COL’s cap” International recognition – not easy without this programme – positive word spreads about COL

Manager

1/ Alignment: Unit was formed to align with the 3 year plan – learning for development for life 3 pillars: education; learning for life; human development

Staff: Manager, co-ordinator and assistant + rental fees self sustainable
Evolved from not generating revenue Patrick Julian COL International –never won the support of the board opposed cross-subsidising – must be cost recovery
Servant of commonwealth NOT competitor

Unit only bids on contracts at UN (development and delivery contracts)

Contracts: delivery and development; 
Unit cannot cross-subsidise; must be cost recovery  but uses surplus for enrichment
2% is reserved for monitoring and evaluation – Results- based mgmt.

UNICEF contract for $1.9m is possible- a roll out of course to 15 countries – it will be sole source  to other sector of COL 10% back to IO because of their input in getting the contract

	
Results

Results cannot just be measured by the stated goals. Other opportunities such as possible contracts with UNICEF and South Africa are possible because of the work of the eLIO. These results should also be considered.

Renewal of contracts and attracting new partners is a successful result.

A practical measure of achievement is essential.

The customized curriculum is heavily prized by partners.

Tutors are eseential for the success of the programme.

Why change a programme that is working?

More attention to working with partners on standards for communications could prove advantageous.

Learners are very positive about the course. Still, there is no post-test validation of achievement.

All the stated goals in the business plan have been achieved.

Possible UNICEF and South Africa contracts are an additional result that was not foreseen.

This is the first time that the eLIO has become fully self-sustaining

Kudos for eLIO staff!

Material quality can be improved.
	COL Staff

Outcomes are based on partners needs and IO could not impact the system of the partners.
75% success rate Ss get constant feedback and personal evaluation

“shows COL can DO not just talk”

Need for practical indicators
they can’t compete with other Commonwealth institutions
goals have been exceeded AND because of IO other things are flowing our way
UNICEF want COL to run teacher training programs (afte losing bid for study) 10% of Angela’s salary to be paid $1.9m.
SAfrica Core business beginning

Recognition of COL comes when partners come back and renew and new partners appear based on word-of-mouth within the UN

The fact that the partners are happy and renewing demonstrates success; Reputation is growing with NO marketing 
no impact studies (Ss feel more confident; reduced time to write)

Recognition has been achieved when partners come back to you or new partners come out – confidence of partners shows effectiveness – learner support with prompt responses

Needs pre and post tests of achievement 

Partners

absolutely achieved goals
DE was very effective as further we go the less opportunities it can be global
Tailor-made curriculum specific to policies of org. updated and flexible evolving over time
Quite effective takes you from wherever you are forward meets multiple need levels

Personalized service – extremely professional “whole tank of knowledge behind COL”

“Thrilled we found them” Their special strength is the system for learners and tutors including the administration and training management.
They were flexible training tutors in our culture
Tutors were very good with a few remarks only about inflexibility all dealt very well – good value for money

 eternal challenge of self-paced learning – getting through the material – it was chunked rather than one large exam – problem was how to reach everyone

Our org keeps shifting. It is difficult to keep up with our needs – can’t track them  Not much COL could do differently Continued demand for these courses high demand  high value ;
 targeted correctly BUT no impact study

Benefits: strategic: more uniform equal program; brought a new culture (we are very bureaucratic) We are medical – clear action oriented simple writing is what is needed for results – people don’t access for themselves BUT there has been a positive shift in their writing and editing skills and more confidence – they are providing substantive input

Helps to standardize core functions with the org. with disparate cultural norms
Promotes conciseness
Good design work interactive and inventive not just the flash. Content reflected the context and rigour of  the f2f courses
Orgs don’t pull together in the UN system – this is a first. They mix with other colleagues – “a rare example of sharing resources” – it addressed a concrete learning problem (IMO World Bank UNHCR) not usual to collaborate.
 “I can’t imagine a better bid”
“I love working with COL”
“COL is a really valuable collaborative partner” “high level of professionalism”

Learner

Very well designed course
Tutors were supportive “wonderful”
CD user friendly
Tough course with concepts in another language; difficult communication at times
You can do better – I am better
“A pleasure to do this course”

Manager

8 partners, surpassed goal
+2000 learners; 62% women
Tested all
3000 people have increased confidence in writing BUT no pre and post test. Some anecdotal samples.
. Goals: 1/3 COL- based – we highlight the COL countries
Achieved goals of developing and delivering of elearning even though we have limited control if any – 

UNICEF contract for $1.9m is possible- a roll out of course to 15 countries – it will be sole source  to other sector of COL 10% back to IO because of their input in getting the contract

South Africa considering a major contract.

06 -07 very successful $700 000 
based on  per capita fee : tutors, evaluation, student marked assignments; material production

Admin fee: 1 – 100 learners %5500 ; 100+ $56 per head
1000 learners pays co-ord position. 
25% overhead is paid for Angela’s time – but this is not strategically considered

+1000 volume gives a margin
per capita fee $450 X 1000 = $450 000 (1100 students)
Manager controls $$$. 08 -09 est. $400 000 ok I need $300 000 more.

UN Auditors’ course 2003 needs 3 – 4 yrs to get it done.
COMSEC wants ecourse in debt mgmt

Changes: positive feedback for 1:1 ecoaching arrangement – a critical success factor; 2 weeks for assignments and 1 per week message

Tutors

Excellent support from COL admin.

Consistency is needed in a standard format: eg version control; naming conventions –important to orgs.

Links are not working in many cases
Needs a detailed review and an editor

but big problem is inconsistency in formatting of the content used. It should model good practice, and be tighter and tidier
forms need formatting to be stripped and simple

Reach

Perhaps an unanticipated example of “Reach” identified by COL staff is the need of other COL sections to benefit from the expertise and skills of the eLIO unit. It should be reaching out to other COL units. 

Reaching out to others can also mean using more technology as appropriate.

Partners are convinced that the online course is the only viable way of reaching staff in developing countries or who are traveling.

Tutors see a need for the oganisations to be more proactive in supporting distance learning.	COL Staff

;if core it could be used by other sectors of COL
Lowest common denominator listserv
1 on 1 training from tutors
tight group that is committed that talks with each oteher it would be more difficult if dispersed & more expensive
A very clearly identified market niche

Cost effective (not fully cost recovery –travel has been reduced)- we would be negative if independent
Tutors could be used to train others in CMW –capacity building

Enhance the program : upgrade level of technology not just lowest common denominator How to improve course materials.
Restructure the assessment; Need a 2nd more advanced course
Specific to jobs for a wide range of people – widen offerenigns
Need more peer-to-peer sharing –social networking
Program could include f2f training

Partners

No other way of providing training to field staff for some learners DE is patently superior ( travelers; those with fragmented work assignments)
More individual help CD not web-based not reliant on connectivity

Tutors

potential for a more advanced course in writing
UN orgs need to learn more about how to support distance learners – need a DE culture in orgs

ESL learners are low to intermediate – not strong English skills; need more of a grammar approach
“I trust the needs assessment programme”

Regions

Capacity building is a core mandate of COL.

Peer-to-peer learning/teaching may be possible with regional groups.	
COL Staff
South Africa 250 000 learners could bring IO to the core by building capacity using the Cdn tutors for training. Have apprentice tutors in developing countries

Tutors

regional groupings might help with peer interactions
discussion forum on the specifics of writing also suggested
materials  customized for a specific company
They like the Canadian tutors
	
Resources

eLIO is in an ambiguous position. Is it core to the COL mission or is it extraneous? Could it be both?

There could be technology fixes to the budget tracking problem.

Technology could bring efficiencies.

If French is not required it should be dropped. It is not economical.

The use of Canadian tutors allows for better quality control. But for capacity building in the Commonwealth the eLIO should be looking at using the Canadians to train others.

eLIO is not resourced as a business, so it is not really ready to be treated solely as a business.

Connectivity is a problem for learners that affects the ways in which the section can use technology.

Partners are concerned about who is taking the course within their organization.

Partners want COL to work with them to acclimatize tutors to their systems.

eLIO’s flexibility is considered to be a real asset.

Tutors are always concerned about success rates.

Standards, standards, standards!

Learners are all working and lack the time to complete.

Self-study aided by formative tests could prove beneficial.

Liaisons with partners have been problematic in some cases. It depends on the partner contact person

Tutors are the backbone of the programme. So quality control is essential.

Good instructional design is essential.

Controlling compyright at least through a license is important for future developments. This issue should be addressed firmly with partners.	COL Staff

Change in Pres lead to emphasis on EBF (Extra Budgetary Funding) because of IO work
IO needs more support; it is a stepchild in the COL program – not considered a useful program

costing and tracking: ea. Cohort has a budget; billing across cohorts is problematic
expanding types of delivery and courses specific to countries
possible mixed model considered.

Needs technology to become more efficient – no other big changes needed
Use Survey Monkey; Basecamp ; listervs as lowest common denominator
IO must break even; it cannot be subsidized by COL; 38 governments in COM – their needs are important
Core funding pays salaries; they do not get 100% back; UK govt in particular insists on offsetting their courses.

French: 5 French courses are not efficient; costs not represented; a drain in costs; little money for translations

Canadian tutors for CONTROL they meet at COL HQ
Don’t mess with a successful model
Cdn tutors – based on logistics – they are trained f2f; with more funding she could get other nationalities Cdns have empathy and understanding

Tutors are what makes it; submit and revise

. Unmet needs: Timing of assignments they are all over the place.
Tasks are not clearly numbered

dependent on UN system not enough $$ Is the money being ploughed back into  section??
* Need for development capital –Can’t make money without investment
Seen as a “cash cow” 

Can’t really operate as a micro-business within COL Must expand 
Need effective people at the agencies to work with.
Tech- connectivity – spam filters causing blockages
Learning technology; access – connectivity

Our ability to scale up better than other orgs with only 1 person
Connectivity – field offices or others on special missions
Some orgs allow extension; others don’t

Partners

Institution more senior people need to take course – majority were female support staff.

Need to work more closely with COL on recruiting/training tutors
We need to internally push more to get more learners
Good quality feedback from tutors
Want to expand to proposal writing

We were a difficult client – COL was very flexible
Design and development was very difficult – a particular non-standard approach; Copyflow process  was not smooth; COL’s system was not what we expected; our system was different. It took a while to gel and was eventually successful
Uneven tutor performance – difficult to control; only small number of tutors did not receive a good evaluation
We were a difficult client – COL was very flexible

Not on COL; inability to fund; not in Spanish

Tutors

75% complete: they have a better awareness of what they can do; some improve noticeably
Time on course cut short by trips
Ss need time during working day; too many are not completing. It’s not a language problem – more a time problem

Learners who persist write more naturally with better structure. They focus on the reader
All about email – a difference in efficiency

They need material on how to present a document; many students start with a qualifier; no articles;
numbering
standardize tighten systematize

Biggest problem for learners is time management
They need more on how to organize and write drafts
Commitment of org. is important for the completion rate – org. ethos

More flexibility to focus on real needs  - let ss skip sections if they can pass a pre-test
More self-testing should be made available for self diagnosis
It was difficult getting real-life examples of writing from the orgs. 

Style guides were not sanctioned
relevancy of materials was sometimes problematic.
Delivery: dodgy infrastructure for dispersed learners
Need a discussion board to enhance collaboration

Need a strong staff development unit in the orgs
Initial contacts are problematic
Submitting marked assignments
Need simpler process for regular tutor reports. Is this administrivia

Manager

Tutors fired for low completion rates – they were weak on the admin side.

4. Enhancing: ongoing – done a lot; highly monitored courses
tech changes enhancing look and feel – trying to keep up with technology

No instructional designer; contract out; intense level of customization
content different for each org.; no generic course; time limits

Copyright belongs to org. work for hire; no license; no problem using in other UN orgs.

Learner motivation level depends on org. rules; some have pay back fee – others no charge.
6. Benefits: ongoing learnin resource; Cd or print; change behaviour write for the reader
Reflect on their own strengths and weakenesses.

Relationships

Building the elearning culture with the partners is important.

Partners recognize and appreciae eLIOs flexibility.

Tutors must be pro-active.

1:1 learning

Empathy

Deadlines are essential. They motivate learners to complete.

The dialogue approach

A reasonable reward system in the organizations boosts success rates.

Personalisation.

Word-of-mouth has produced success.	COL Staff

No culture of independent self-directed learning

We’ve been told that evaluations are not enough – stakeholders need them.

Partners

We were a difficult client – COL was very flexible

Tutors

Effectiveness: need tutors or they would get “swept away” by the content. Tutoring is ongoing and pro-active; Ss. Appreciate the concern of the tutors

remarkably effective – close to the customer with customize content a great deal of effort to ensure the fit. Content quality is very high
Tutors are interested in the learners “compassionate, responsive, individualized”

Deadlines are to learners who are supposed to set them with tutors. Much leniency except when org require end date. Tutor nags them

As student follows through on assignments, “it becomes a dialogue” assignments feedback improvement
“Those who understand it as a process take off” – get kudos at work – others don’t learn as much
Orgs desperately need this training .  The Ss are highly academic – write academic papers with much verbiage 50cent words, passive tense. They need plain English

Boost in confidence; courses recognized for promotion; significant improvement in writing; DE skills are very important
External tutors are not involved in org. politics
Appeal of 1:1 tutoring makes it more intimate

Bank of tutors, trained and motivated

For many it was the first time that someone looked at their writing as writing; someone outside the field with pointed feedback

Manager

Goals: 1/3 COL- based – we highlight the COL countries
Achieved goals of developing and delivering of elearning even though we have limited control if any – 8 partners, surpassed goal
WHO         UAC Refugees
ILO             Red Cross 
UNAIDS    World Bank
UNICEF     OIOS (auditors)
	
Comments on Manager

Would this section survive without the present Manager?	COL Staff

Angela and her unit is seen as an “ugly duckling” where an atypical role has evolved; 
Angela is needed. We would not know how to administer and support this unit

Angela: Person-based IO; very efficient; if she left we would review;
She runs a tight ship.	

Annex B

Survey Monkey

http://www.surveymonkey.com/MySurvey_EditorPage.aspx?sm=pGkjPpQxV2OfrYhq8wLJRBxppn8h13mBe9E8ehMNdNU%3d

Page: COL Student course evaluation

1. My writing and analytical skills improved significantly due to taking this course.
 	Very high	Significant improvement	Marginal improvement	No improvement	
Improvement 
level	10.0% (1)		50.0% (5)		30.0% (3)		10.0% (1)	

	 
2. The course tasks and assignments were appropriate and linked to the course objectives.
 	Very appropriate		Somewhat appropriate		Not appropriate	
Appropriateness	60.0% (6)		40.0% (4)			0% (0)	

3. How clear and easy to understand were the directions and assignments?
 	Very clear	Satisfactory	Needs improvement	Unclear	
Materials 
clarity	50.0% (5)	30.0% (3)	20.0% (2)		0% (0)	

4. Was the number of assignments appropriate? How difficult did you find the assignments?
 	Too many or too difficult	Appropriate for course	Too few or too easy	
Number of 
Assignments    10.0% (1)		90.0% (9)			   0% (0)	
Assignments 
difficulty	0% (0)		88.9% (8)			11.1% (1)	
 	
5. How many times did the tutor contact you?
 		   	  +12	     	 +6		+3		1	0	
Primarily for 
administrative help  	0% (0)   		55.6% (5)    	44.4% (4)	0% (0)	0% (0)	
Primarily for course 
content help	  	10.0% (1)  	40.0% (4)   	50.0% (5)	0% (0)	0% (0)	

6. How many times did you contact the tutor?
 			+12		+6		+3		1		0	
Primarily for general 
information:		0.0% (0)		11.1% (1)	77.8% (7)	11.1% (1)	0% (0)	
Primarily for 
help with 
course content:		12.5% (1)	25.0% (2)	50.0% (4)	12.5% (1)	0% (0)	

7. How important would you rate the interactivity with the tutor?
 	     	Essential		Important	OK	Unimportant	Not really needed	
Interactivity  	40.0% (4)		60.0% (6)	0% (0)	   0% (0)	   	      0% (0)	

8. Could you have completed this course independently without the tutor?
 	    Essential	 Extremely helpful	      Helpful	A little helpful	Not really needed	

Tutor is 
needed	    20.0% (2)	      50.0% (5)	     	20.0% (2)	10.0% (1)	0% (0)	

9. Please add any comments below that you wish to make about the course or about this survey.
 
1.	very useful	 Fri, 6/13/08 11:31 PM 
	
 2.	I couldn't complete the course, the deadlines were too short regarding my work load	 Thu, 6/12/08 9:40 AM 
	
3.	Instead of having instructions at each level Pre-writing, organising, drafting, revising and proofing reading it would help if there were just one set of instructions to go through. It saves time. Secondly the course content is too much and impossible to complete within the deadline. Thirdly, there should be flexibility of time for resubmissions i.e. it should not be linked to the deadline for submission of assignments. Having a seperate deadline provides the learner an opportunity re-visit the modules to gain a better understanding of the subject.	 Wed, 6/11/08 10:20 PM
 
4.	It was a very interesting course. Since I am not a native English speaker, I think the course was toughter for me.	 Wed, 6/4/08 8:57 AM 
5.	As the course was a pilot, there were undoubtedly a number of improvements required. Above all, insufficient time frame, lack of clear section markers, lack of initial clarity that the course was to be done concurrent with a professional activity to enhance its usefulness and in order to accomplish the objectives, lack of variety of examples and contextualisation, lack of clear instructions for some of the activities and alternative answer keys. In terms of Question 1 above, I found slightly more than marginal improvement in my analytical skills from taking the course, although it was mainly related to the specific course content and nature of the course, as I have already attained an advanced writing level academically.	 Wed, 6/4/08 3:19 AM 

6.	The role of the tutor makes the course different. Personal advice received gave me much more than I expected. He was also encouraging me well. The biggest difficulties faced was in the beginning, when the timetable was unclear and not easy to find and understand.	 Wed, 6/4/08 2:40 AM 

7.	Completing the course while on job requires a longer time period as job priorities and deadlines compete with course deadlines

Annex C

Consent Agreement

The Commonwealth of Learning has commissioned Professor Rory McGreal, Associate Vice President of Athabasca University – Canada’s Open University to conduct an evaluation of the COL International Organizations section.  

We are requesting your informed consent to participate in an interview either via email, face-to-face or by telephone. This should take no more than 30 minutes.  Dr McGreal will be contacting you shortly requesting your participation. Your participation in this research is entirely voluntary. You may refuse to participate or may withdraw from the interview at any time.

You or your organization has been chosen to participate either by random sampling or because your knowledge is particularly relevant for inclusion in the evaluation. We expect to interview up to twenty people in this process. The identities of the people providing individual responses will not be made available, and will not be listed on any documentation.

Risks:
As the researcher will be interviewing people who are known to the staff of the International Organization of COL, there is a small risk that you could be identified by your comments.  Identification of interviewees may be made by category, for example, tutors, staff member, student. There is no direct benefit to you as a participant. The researcher intends to use this internal report as the basis of an article for submission to a scholarly journal and for presentation at a scientific conference. Participants’ names will not be used, nor made available.

Privacy/Confidentiality
The names of participants will not be recorded. Access to the data will not be made available to COL staff members.  The records will be destroyed after three years. Participants will be informed when the final report is submitted showing results of the research.
    
The researcher can be contacted at rory@athabascau.ca  or 780 994 9579.

Annex D

COL evaluation questions for staff

Preliminary:
	Position: 
	Sex:
	Yrs. In position:
	Role with eIO COL: limited	marginal	ongoing	intensive

1. Could you provide me with an example of the eIO unit’s alignment with COL’s goals?

	5			4			3			2			1
Strong alignment									   Weak alignment

2. What direction changes did the unit make as a result of previous evaluations? Were these positive or negative?

	5			4			3			2			1
Positive								   			Negative

3. Has COL policy zigzagged? Alternated? Expensive projects? Cutbacks? How has the unit coped with this?

	5			4			3			2			1
Positive								   			Negative

4. Did the COL eIO achieve its goals? Namely:
o	Development and delivery of technology mediated distance learning programmes with international organisations to enable access to learning by a dispersed and diverse international workforce.

	5			4			3			2			1
Surpassed expectations									    Failed

o	By 2009, at least 6 major international organisations active in delivering the MDGs have formed a partnership with COL in customising gender and culture sensitive workplace eLearning programmes.  

+6    		5		4		3		2			1
	Surpassed goal			Organizational participation   		   Failed

o	An additional 2,000 adult learners (a 6 to 4 ratio of women and men) will be trained by 2009.

+2 000			+1500					less than 1000	   
	Surpassed goal				Sex ration				Failed

+6:4			6:4			less than 6:4 > 1:1		less than 1:1
Surpassed goal 						   			Failed

o	At least two new contextually relevant sets of ODL/ICT-based learning materials for occupational skills development created and each is taken up by at least two major institutions.

+2			2						2			1
Positive					Materials		   			Negative

6. Did the COL eIO achieve its goals through the activities posited? Activities:

o	Actions to achieve the outputs
d.	Analyse and understand the learners’ needs, learning objectives and outcomes, the learning environment and technology infrastructure of the organizations

	5			4			3			2			1
Positive								   			Negative

 

e.	Design courses and develop materials through customization

	5			4			3			2			1
Positive								   			Negative

f.	Test all training materials and delivery designs

i.	Build responsive infrastructure and clear processes for course production and presentation. This includes: 
			* preparing storyboards/templates;
* using robust, affordable and user-friendly technology and     
   media;
 	* recruiting and maintaining a cadre of trained and        				   committed tutors;
	* using OERs software as far as possible

o	Monitoring Data
vi.	pilot evaluation reports
vii.	monthly progress reports
viii.	tutor listservs
ix.	end of course evaluation feedback
x.	programme evaluation studies

Questions for partners and tutors

1.	What is your current role in your institution?

2.	Can you discuss the effectiveness of the eIO COL training programme?

3.	In your view, what unmet needs do your learners have that could possibly be addressed via changes to the programme?

4.	What suggestions do you have for enhancing the current training programme (ie. what can we be doing differently)?

5.	What would you identify as some of the major roadblocks that were encountered  (if  any) in implementing and sustaining the programme? 

6.	What would you identify as some of the benefits of providing staff with this online programme?

7.	How could the programme be rendered scalable? Development (customisation – how much is needed?) Delivery?

8.	On a scale of one to ten, with one being not important and ten being very important, how important is it to provide distance education opportunities to your staff to supplement or replace any other training that you provide.

Annex E
Terms of Reference

Consultant responsibilities
1.1. The Consultant shall:
1.1.1. review the draft evaluation plan and relevant COL documents related to the evaluation activity;
1.1.2. assist in the identification of appropriate data to be collected and in the development of data collection strategies, particularly in the areas of :
1.1.2.1. Growth of the initiative 
1.1.2.2. Learners, training chiefs, supervisors and partners satisfaction 
1.1.2.3. Consistency with COL mission 
1.1.2.4. Cost effectiveness
1.1.3. implement the evaluation plan;
1.1.4. visit COL and conduct interviews with key partners, learners, tutors and COL staff, particularly in the areas where we have helped partners to achieve;
1.1.5. assess the quality, outcome and output of the initiative based on the 6R assessment criteria (Relevance, Results, Reach, Regions, Resources and Relationships); and
1.1.6. produce an evaluation report according to the attached Evaluation Report Format.
2.0 COL Responsibilities
2.1. COL shall:
2.1.1. provide appropriate briefings and coordinate with stakeholders concerning the evaluation;
2.1.2. provide access to reports and other COL documentation relevant to the activity; and
2.1.3. provide feedback to the Consultant on the draft report for finalization by August 31, 2008.
2.2. The COL staff member with whom you should liaise on all matters related to this evaluation is Angela Kwan.

Annex F
	
Contributor anonymity statement

As expressed in the consent agreement, the names of participants have not been recorded. Access to the original data will not be made available to COL staff members.  The records will be destroyed after three years. 

Annex G
Reports and publications consulted
Committee on Benefits of DOE R&D on Energy Efficiency and Fossil Energy. (2001). Energy Research at DOE: Was It Worth It? Energy
Efficiency and Fossil Energy Research 1978 to 2000. Washiington DC: Commission on Engineering and Technical Systems, National Research Council.
Commonwealth of Learning. (1999). Final proposal (UNHCR bid). Vancouver: Commonwealth of Learning.
Commonwealth of Learning. (2002a). Technical proposal (AIDS/WHO). Vancouver: Commonwealth of Learning.
Commonwealth of Learning. (2002b). Technical proposal for Writing Effectively: A Submission to  the International Labour Organization Vancouver: Commonwealth of Learning.
Commonwealth of Learning. (2006). Learning for development: The 2006 -2009 Plan, Retrieved from http://www.col.org/colweb/site/pid/3939
Commonwealth of Learning. (2007a). Evaluation request for baseline information. Vancouver: Author.
Commonwealth of Learning. (2007b). Completion Report: WHO/UNAIDS Cohort #12: June 15 – December 15, 2007. Vancouver: Commonwealth of Learning.
Commonwealth of Learning. (2007c). L4L AK Elearninig with IOs: Evaluation plan outline. Vancouver: Commonwealth of Learning.
Commonwealth of Learning. (2007d). Results for WHO June 2007 session. Vancouver: Commonwealth of Learning.
Commonwealth of Learning. (2007e). Submission to CADE Awards of Excellence Programs: “Writing Effectively for UNHCR” course Vancouver: Commonwealth of Learning.
Commonwealth of Learning. (2007f). Technical proposal for Writing Effectively for OIOS:  A Submission  to the United Nations Vancouver: Commonwealth of Learning.
Commonwealth of Learning. (2007g). The World Bank HRSLO: Report writing (CDRom). Vancouver: Author.
Commonwealth of Learning. (2007h). World Health Organization: Writing effectively for WHO (CDRom). Vancouver: Author.
Commonwealth of Learning. (2007i). Writing effectively for UNICEF:  A Submission to UNICEF (India) Vancouver: Commonwealth of Learning.
Commonwealth of Learning. (2008a). Commonwealth of Learning discussion area: List of listservs. Vancouver: Author.
Commonwealth of Learning. (2008b). International Labour Organization: Writing effectively for the ILO (CDRom). Vancouver: Author.
Commonwealth of Learning. (2008c). Listserv participation list. Vancouver: Author.
Commonwealth of Learning. (2008d). Monthly progress reports (WHO, UNAIDS ILO, ILO2, Worldbank). Vancouver: Author.
Commonwealth of Learning. (2008e). Pilot: Writing effectively for UNICEF, India country office (CDRom). Vancouver: Author.
Commonwealth of Learning. (2008f). The World Bank HRSLO: Writing e-mails, memos and letters (CDRom). Vancouver: Author.
Commonwealth of Learning. (2008g). OIOS course analysis. Vancouver: Author.
Commonwealth of Learning. (n. d.-a). Draft only evaluation report: Writing Effectively for UNHCR (Trial Version) Vancouver: Commonwealth of Learning.
Commonwealth of Learning. (n. d.-b). Evaluation request for baseline info. Vancouver: Author.
Commonwealth of Learning. (n. d.-c). Proposal for IFRC. Vancouver: Commonwealth of Learning.
Commonwealth of Learning. (n. d.-d). Writing audit reports for OIOS - Development cost analysis. Vancouver: Author.
Dunlop, C. (2005). Writing effectively - Distance learning course for the International Federation of Red Cross and Red Crescent Societies:  Assessing the return on expectations Vancouver: Research and Evaluation Unit, Simon Fraser University 
John, K. (2005). Writing Effectively for UNHCR:  Course Evaluation Report Vancouver: Commonwealth of Learning.
Kwan, A. (2002). Staff development: training more with less – a model for international development agencies Pan-Commonwealth Forum 2. Durban, South Africa.
Kwan, A. (2005a). Comparing learning effectiveness based on use of different media for delivery of content Pan-Commonwealth Forum 3. Dunedin, South Africa: Commonwealth of Learning.
Kwan, A. (2005b). eLearning for International Organisations. Vancouver: Commonwealth of Learning.
Kwan, A. (2005c). Staff development : Training more with less:  A model for international development agencies, Pan-Commonwealth Forum 2. Durban South Africa: Commonwealth of Learning.
Kwan, A. (2006). Business plan for COL programme – “Human resources development through eLearning for international organisations”  2006-2009:Submitted to senior management meeting, April 2006 Vancouver: Commonwealth of Learning.
Kwan, A. (2008a). Learning for livelihood: Transnational programmes. Vancouver: Commonwealth of Learning.
Kwan, A. (2008b). Learning for livelihoods: Transnational programme (Powerpoint). Vancouver: Commonwealth of Learning.
Kwan, A., & Martel, A. (2005). Designing distance education for international organisations: Testing theory from a field-attentive approach, Canadian Association of Distance Education Conference. Vancouver.
Kwan, A., & Smulders, G. (2006). Think globally, learn locally:  Democratising learning for international organisations, Pan-Commonwealth Forum 4. Ocho Rios, Jamaica.
Lentell, H., & O’Rourke, J. Tutoring large numbers: An unmet challenge International Journal of Research in Open and Distance Learning (Vol. 5).
Murphy, D. (2003). Evaluation report of a course on Writing Effectively: Commonwealth of Learning/ International Federation of Red Cross and Red Crescent Societies Hong Kong: Open University of Hong Kong.
O'Rourke, J. (2003). Lost and found: Elearning outside the doors of academe. In A. Tait & R. Miller (Eds.), Rethinking learner support in distance education: Change and continuity in an international context. London: Routledge.
Ruhe, V. (2002). A Course on writing effectively for UNHCR: Evaluation report  Vancouver: Commonwealth of Learning.
Ruhe, V., & Zumbo, B. D. (2006). Using Messick’s Framework to validate assessment tasks in online environments:  A course in writing effectively for UNHCR. In D. D. Williams, S. L. Howell & M. Hricko (Eds.), Online assessment, measurement and evaluation: Emerging practices, (pp. 203-226). Pennsylvania, Information Science Publishing, Idea Group Inc.. (pp. 203 - 226). Pennsylvania: Idea Group Inc.
Shale, D. (2002). Evaluation report on WHO/UNAIDS training course - Writing Effectively (Pilot offering September/October, 2002). Calgary: University of Calgary.
Spaven, P. (2006). Evaluation of the 2003-2006 Commonwealth of Learning Strategic Plan: Final Report. Retrieved from http://www.col.org/colweb/site/pid/3939
Trevor-Smith, H. (2008). Transnational programmes (Powerpoint). Vancouver: Commonwealth of Learning.
Walker, R., & Dzakiria, H. (2004). Tutor's feedback on the course on Writing Effectively for UNHCR/UNAIDS and WHO: Stage 2. Norwich, UK: University of East Anglia.
Walker, R., & Dzakiria, H. (2005). Impact of the Effective Writing Course in WHO and UNAIDS: Evaluation Study for the Commonwealth of Learning - Stage 3. Norwich, UK: University of East Anglia.
Wilson, M. (2006). Evaluation report : Writing effectively for the ILO Vancouver: Commonwealth of Learning.
Wilson, M. (2007). World Bank Report Writing distance learning course: Evaluation Report. Vancouver: Commonwealth of Learning.