The evaluation of learning is a critical issue, for if we don’t evaluate then we are not measuring whether learning has occurred.

I am currently commencing a project – within a large organisation with branches in most States and Territories – to assist its learning evaluators to develop new skills. There is enough time available for the project to assist participants from around Australia to develop skills and knowledge not just in the basics of evaluating learners’ reactions and whether they acquire new skills and knowledge, but also to assist participants to measure whether the learning affects job and organisational performance.

This is a rare opportunity to work with learning evaluators over an extended period time, as many educators and learning and development managers are focused narrowly only on immediate learning and questions such as: Did participants benefit from the face to face instruction? Can they demonstrate a new skill? Can they articulate new understanding?  However, there are many more challenging questions that can be asked three-six months after a training event, such as: Are participants able to demonstrate they still have the initial competency? Can we identify any benefits for the organisation from the training program? What was the return on the investment in training?

For learning evaluators to be able to learn to measure the long-term effects of training, I find a comprehensive professional development program is required, which delves into the deep layers of evaluation. These deeper layers involve developing knowledge and skills in areas such as: 

• developing an evaluation framework for any training program
• designing and implementing evaluation tools
• analysing and reporting on the data, for multiple audiences.

The forthcoming professional development program will also create the conditions for participants to share their experiences and learn from their peers, in an atmosphere of trust and goodwill. This collaborative approach is vital, as there is so much for each individual to learn about evaluation, and learning from peers is invaluable. One primary participants will learn is that evaluation is an art as well as a science, as evaluators are commonly required to adapt tools and strategies to fit the culture and setting of an evaluation. Evaluation requires the exercise of professional judgment and sensitivity.

This need to exercise judgment will be of concern to those who believe that evaluation tools can be robotically pulled down from a website, without any modifications. Making the modifications to existing templates is just one of the high-level skills required in an effective approach to evaluation.

Tomorrow, Thursday 14 October 2004, I am presenting a paper - as lead author - at the Australasian Evaluation Association’s Annual Conference in Adelaide. The paper is on comparative methodologies to measure the return on investment on training.

One of the methodologies I will report on is that of assessing the value for money. The Australian National Audit Office (ANAO 2002) defines value for money as a judgement of the worth of funds expended in the light of the benefits received (p.84). Types of VFM evaluation types noted by the ANAO include:

• cost comparisons between internal programs and those available commercially;
• benchmarking against other agencies;
• assessment of value for money based on participant and supervisor feedback. (p.85)

The potential benefits of VFM studies include the quick feedback they provide to decision makers and the public demonstration that the expenditure of funds is being evaluated using comparative data.

My study shows that value for money studies are straightforward to undertake compared to ROI and ROTI studies and they yield important data. This finding supports the ANAO (2002) which recommends the frequent use of VFM studies.

Today I drafted a paper - as lead author for a conference paper - on measuring the return on investment (ROI) of a leadership program. The paper raises the issue that conventional ROI studies do not suit many educational settings.

There is a growing interest in Australia in calculating the return on investment (ROI) from training. However, ROI studies suit training which has easily quantified outcomes, like how many more cars did a car salesman sell after attending a crash course on how to sell cars. ROI studies are fine in principle but may not suit all contexts.

The interest driving ROI studies of training is admirable. Shandler (1996, p.101) notes that the increased interest in measuring the costs and benefits of training is due the paradigm shift in training over the last ten years. Training is often now linked to specific business needs and it addresses specific objectives. In a time when new business needs are being driven by global competition and new technologies, accountability of all business functions is increasing. This new corporate reality requires an organisational mindset that measures changes in business results attributable to training.

Phillips (1997) suggests that there are four theoretical benefits of an ROI study, all of which are worthy pursuits:

• to measure the contribution of specific programs to corporate objectives and in that process, to determine if the benefits (expressed in monetary terms), outweigh the costs
• to enable the setting of priorities, based on a program’s level of contribution to meeting corporate objectives
• to enable a focus on results
• can alter management perceptions that training is an investment and not an expense.

Pepper and Christie (2000) define the ROI process as showing the bottom-line impact of training and development programs and corporate initiatives. ROI also serves as a tool to demonstrate accountability within the organisation.

However, the term return on investment originates from the finance and accounting field (Moy in Smith, ed. 2001), so it is easier to apply the concept to a professional development program that leads to quantifiable outcomes such as an increase in sales than to professional development programs that lead to improved leadership or interpersonal skills.

In the ROI study of a major Australian agency’s leadership program which I am reporting on in a conference paper, I discuss how the ROI study was unable to highlight properly the qualitative findings from the program. To compensate, I used some other strategies.

In my work as an evaluator, I use a flexible approach to ROI. I also use several alternative strategies. Education is such a large industry, it does not have to use an ROI model built for the manufacturing and other industries: it deserves and can have its own variations of ROI frameworks.

In an earlier posting I reported on an evaluation I conducted of the British Broadcasting Corporation (BBC, UK) trainee schemes in early 2004. As part of that evaluation, a benchmarking activity was undertaken with the Australian Broadcasting Corporation (ABC), comparing its experiences with trainee schemes with the BBC’s experiences. Some points from the study of the ABC follow.

Over the last two decades the ABC has conducted entry-level workbased trainee schemes, combining paid work and structured training, irregularly: the last major intake was in the mid 1980s, when a total of around 20 operational trainees and 10 technical trainees were taken on in TV and radio. Smaller intakes have been accepted since then and graduate-level trainee schemes have been conducted even less consistently.

Jenny Ferber, Head of Training at the ABC, and who approved this posting, has encouraged discussion of the issues around trainees within the ABC, originally tabling a discussion paper. Currently she is leading the development of a new policy on the matter. The new policy is expected to suggest more efficient processes, such as common reporting and linking all trainee schemes, where possible, to the national qualifications.

Jenny Ferber is supportive of national qualifications for a number of reasons, including the fact that they ensure a minimum but satisfactory and consistent standard. However, she will withdraw her support if the national qualifications framework, particularly Training Packages, fails to ‘keep up’ with changes in the industry.

Key findings from the benchmarking included the following:

• There are many more similarities between the ABC and BBC trainee schemes than differences.  Key similarities include the needs such as replacing existing staff; hindrances such as a lack of jobs for every trainee; a commitment to diversity; and the use of internal accreditation.
• Some differences between the ABC and the BBC include the stronger commitment to national qualifications by the ABC – provided the Australian national training framework remains current.
• The ABC is undertaking a review of its trainee schemes in 2004 and will produce a new policy paper shortly. Like the BBC, the ABC intends that its new policy will clarify the value of different types of schemes and will ensure consistently high standards of outputs, partly through central coordination of some aspects of the schemes.

Given the way it is continually seeking to improve itself, the ABC meets the criterion for a high-performing organisation that I identified in relation to the BBC: humility. In the case of the ABC, humility means continually seeking to improve systems and impacts.

This study reinforces a number of other issues current in VET:

• The BBC is benchmarking with the ABC because the UK and Australia are both part of the global economy.
• For Australian companies to be on the pace internationally, our training needs to be world-class.
• Training in the ABC is comparable in quality to the BBC, which is encouraging for Australian VET, but the ABC, like the BBC, is seeking to improve its training even further. The high jump bar for Australian training keeps rising.

In a recent blog entry I highlighted the emergence of a new characteristic of high-performing organisations: humility. 

I take humility to mean a willingness to listen to staff and clients and to review all current and past structures and assumptions. Humble, high-performing organisations willingly embrace new ideas, new approaches, new systems and new relationships. People guiding high-performing organisations continually learn and are agile in their response to their new learning.

I saw an excellent example of organisational humility when I undertook an assignment for the BBC in London earlier this year. My task was to frame up and commence an evaluation of the long-cherished in-house training within the BBC.

The corporation has delivered trainee schemes for several decades and spends around AUD$25m per annum on these schemes and a total of AUD$125 pa on training and development. Although the claim that one was ‘BBC trained’ is respected around the world, the BBC wanted a thorough analysis undertaken of BBC internal training. Humility at work.

In interviewing a range of BBC trainees, both current and past, and a range of staff, I was struck by a number of responses to the evaluation:

• the desire of BBC management that the existing trainee schemes be benchmarked against the best in the world, to make sure the BBC was still on the pace
• the determination of BBC management that the trainee schemes be critiqued to make sure there were consistently high standards of training delivery
• the attitude of interviewees that the reputation of the BBC trainee schemes could only be maintained by continuous improvement, not clinging on to past glories. 

For me, the BBC is high-performing because of such humility.

Measuring the value of training is an elusive goal, unless considerable rigour is brought to the exercise.

I am the lead presenter of a paper on this topic at the Australasian Evaluation Society 2004 International Conference, Adelaide, 14 October 2004 www.aes.asn.au . The title of the paper is 'The comparative value of three different methodologies for measuring the return on investment from a leadership program'.

The paper is based on an evaluation I conducted of a major leadership program within a large government agency in 2003. The evaluation consisted of three separate studies: a sample Return on Investment (ROI) study, a sample Return on Training Investment (ROTI) study and a sample Value for Money (VFM) study. The paper reports on the comparative value of each of these three different approaches.

The three studies focused on the pilot group who participated in a residential program in March-April 2003 and follow-up activities, including participation in coaching and peer partnering. In addition to other data collection, six of the twenty participants in the pilot program were interviewed after three months and seven were surveyed after three and six months. To ensure validity, seven colleagues of the seven participants were also interviewed, to check their assessment of the impact of the leadership program on the participants.

The ROI study investigated whether the pilot program generated returns which outweighed the costs. The ROTI study focused on a small number of expected outcomes and related performance measures. The VFM study investigated whether the leadership program provided the organisation with value for its outlay.

In addition to making a range of recommendations, the overall study highlighted the different nature and benefits of each approach. This comparison of a diversity of evaluation methodologies was designed to assist the organisation's staff to make informed decisions in future as to whether they use any one or more of these three approaches.