Key challenges

In developing a methodology for evaluating researcher development activity a number of key issues need to be addressed.

Specific evaluation issues for the researcher development sector:

1. Demonstrating the value of the researcher development agenda to stakeholders; researchers, government, funders of research, higher education institutions,  supervisors, employers of researchers etc.
2. Articulating and presenting evidence in a format appropriate for each stakeholder.
3. Existing evaluation models tend to be developed for the commercial sector, rather than higher education, and tend to focus on return on investment (ROI).  ROI is increasingly important in higher education, but associated timescales can be significant. More critical initially is the evaluation of learning and behaviour changes.
4. There are many techniques employed for assessing learning in higher education however, where a programme is not compulsory (common for researcher development) it is more difficult to apply e.g. examination or course work type assessment.
5. It is only recently that there has been a significant growth in researcher development activity and related evaluation. There is no significant body of prior work to draw upon.

Challenges common to any training and development evaluation:

1. The issue of attribution - there are many potential influences on the learning and development of individuals. How can you be sure that any achievements in development of an individual are in some way attributable to a training activity and not some other external influence?
2. The subjective nature of participant views - participants may report that they have learnt from a development activity. How can you confirm that they actually have?
3. The potential of metrics to mislead - metrics quoted in isolation without reference to or knowledge of the ‘story' behind the number are open to misinterpretation.  Metrics such as increasing numbers of participants in training and development activity are valuable, but they don't indicate learning or changes in behaviour.

How does the Rugby Team Impact Framework and implementation address the key challenges?

The key challenges are addressed in a number of ways, however, the mains points are:

1 - 2 The impact framework and implementation has engaged a wide range of contributors providing evidence of a diverse nature in building an evidence base. This provides a body of evidence from which stakeholders can draw to meet their own needs. The ‘language' of the impact framework levels allows the ‘translation' of evidence gathered through a multiplicity of methods to be commonly accessible to stakeholders.

3-5 The Impact and Evaluation Group was formed as a sector working group and developed an impact framework tailored for the needs of the sector and stakeholders in researcher development.

6 The problem of attribution -

• Within specific evaluations the same cohort of participants are tracked through the impact levels.
• A body of evidence has been built nationally through higher education institutions sharing evaluation data and the mapping of evaluations against the RTIF. Any conclusions about attribution can therefore be drawn from data collated from multiply studies.

7 The subjective nature of participant views - A mixed methods approach to evaluation is commonly used i.e. - interviews, focus groups, and a range of other outcome measures. Therefore conclusions can be drawn from a range of data sources not just participant views.

8. Metrics - metric data is only part of the data collected and is enhanced by data collected at levels 1- 4. Evaluation outcomes are presented in collated reports detailing a wide range of data sources (see the ‘Rugby Team Impact Framework: one year on')