C3/D3 Workshop summary and outcomes

Measuring the impact of skills development

Mark Beatson, Science and Innovation Analysis, DIUS and Dr Tony Bromley, Senior Training and Development Officer, University of Leeds and Vitae Yorkshire and North East Hub Co-ordinator

Overview

Investment in training and development activity for researchers is one aspect of investment in research as a whole. This workshop explored the ‘big picture' of research investment, putting into context and highlighting the importance of measuring the impact of the skills development component on policy making and how this relates to all stakeholders.

The session covered:

• the government perspective, including:
  • looking at evidence useful for policy making
  • the benefits of having evidence on researcher development to feed into broader thinking in investment in research (leading to the next Comprehensive Spending Review (CSR))
  • why HE institutions should engage in evidence building
• the perspective of higher education institutions and researchers, including
  • how the Rugby Team Impact Framework can support the development of a robust evidence base
  • how the Rugby Team Impact Framework can engage stakeholders in the evidence agenda
  • the University of Leeds' experience to date in implementing the Rugby Team Impact Framework.

Presentation summary

A Government perspective: Mark Beatson

Top-level questions in government spending are:

• how much public money to spend on funding research through the Science Budget (Why is public investment needed? What are the economic and social benefits from different expenditure choices?)
• how should this budget be allocated between different uses? (between parts of the dual funding system, research councils, modes of funding)
• what is the balance of funding for future researchers between quantity (numbers of posts) and quality (researcher support and development)?

The Economic Impact Reporting Framework is used to:

• report on UK progress against ten-year framework objectives
• organise output and performance data from research councils
• organise disparate material.

Current human capital indicators are STEM (science, technology and mathematics) A Levels, STEM and non-STEM graduates (including by area of study, gender, ethnicity) doctoral graduatess: numbers, per 1000 population (and by area of study) and researchers per 1000 of the workforce.

Various types of evidence are needed by the Research Councils to support future decisions:

• reliable data on outputs
• link through to outcomes (economic and social) net of cost of investment
• evidence of value for money
• quantitative and qualitative.

Evidence can come from administrative data and follow-up surveys of researchers. ‘Impact frameworks are about how we bring some consistency to the interpretation and design of evidence gathering'.

Persuasive evidence (in this policy context):

• takes time to put together
• places strong emphasis on quantification, though qualitative information has its place
• remembers the basics (eg what did the money buy?).

View Mark Beatson's presentation.

Measuring the Impact of Skills Development - an HE institution and Researcher Perspective: Tony Bromley

The main themes highlighted were building on means to find answers to the questions:

1) How can the Rugby Team Impact Framework support the development of a robust evidence base?

2) How can it engage stakeholders in the evidence agenda?

Tony described the theoretical basis and practical application of the approach taken at the University of Leeds as an example of developing the Rugby Team Impact Framework in institutional practice.

A further update/report will be provided to the January 09 Roberts Policy Forum including examples of HE institution evaluation practices (eg from the Vitae Database of Practice). 

View Tony Bromley's presentation.

Discussion outcomes

Participants concurred with the key messages:

• Be careful what you evaluate (what is its purpose and value? Have you clearly identified the need?)
• Be patient for impact.

Discussion also identified a number of problems with assessing impact (at various levels, such as the issues associated with the length of time required to gather outcomes data (more than three years up to ten years or more) and the challenges of understanding the context in which learning is being applied. Some participants saw obtaining high level outcomes data as very challenging.

It was felt that the problem of attributing causality (training/development being only one of a number of influences on the researcher) could be addressed by doing multiple evaluations of an aspect not just one form of data collection.

Participants were encouraged to share their evaluation practices.

The Rugby Team Impact Framework project was asked to clarify how Vitae might help aggregate and feed evaluation work being undertaken in HE institutions through to the Research Councils to be fed into policy decisions.

 

View the C3/D3 presentation slides for Mark Beatson
View the C3/D3 presentation slides for Dr Tony Bromley

View Dr Tony Bromley's handout