UCL Impact Assessment Project – Personal and Professional Management Skills

What is the rationale for doing this?

How does it fit with institutional strategy?

What are the main features of the provision?

What are the aims and expected outcomes?

In response to the expectation from the Research Councils that institutions undertake impact assessment in relation to the uses of the Roberts funding, at the higher levels as defined by the Rugby Team Impact Framework, the UCL Graduate School initiated three projects to look at the impact of key elements of our training provision.  This current DOP entry describes the impact assessment project  for UCL’s "Maximising your Potential" personal and professional development programme for research students - Programme Director and impact project leader Dr Paul Walker from UCL's Centre for the Advancement of Learning and Teaching.  For information on the other two projects (looking at Career Development, and Entrepreneurship) , please see DOP entries 953 and 1214 respectively.

The findings of the impact assessment projects have been used to inform our own provision and contribute to the national debate on the impact of researcher development training.

A number of the courses provided by CALT for UCL research degree candidates via the Graduate School programme are organised into a four-part programme dubbed ‘Maximising Your Potential’. Synoptic information on the courses comprising this programme is appended herewith. The heart of the programme is a 3-day residential course, focusing on self-directed skills development, with opportunities for participants to work on a variety of generic skill areas under priorities indicated by development needs analysis. These issues are also touched on in Part 1 of the programme, a one-day workshop, which explores development needs analysis, using the UCL Graduate Research Student  Log, as well as guidance for conducting supervisory meetings.

Students are advised that they may progress through the four-part programme in sequence, or to select those courses which they consider to be potentially beneficial. A particular feature of this programme is the opportunity for participants to train as facilitators, with subsequent opportunities to work with the course convenors in supporting the learning and development of others in workshops and residentials. In practice, the first two courses in the programme appear to serve different needs and interests, since overlap in participant cohorts is more the exception than the rule. This may be due to their quite different aims, formats and duration. More advanced parts of the programme recruit mainly from the residential courses directly.

Each course is evaluated within a consistent framework with data obtained from both participants and members of the staff/facilitator team. The outcomes of these evaluations are generally quite positive, with upward trends observed in key metrics over successive years. Occasionally outliers in the data are observed, usually associated with students being ‘sent’ to the courses by their departments, despite our advice to the contrary. A summary of participant evaluation data for the most recent annual cycle of residential courses is appended herewith.

Post-course evaluation such as described above offers insight into impact at levels 1 & 2 (‘reaction’ and ‘learning’ in the Vitae Guide to Evaluation). Anecdotal evidence suggests unsurprisingly that engagement with the programme over an extended period has greater impact at the deeper levels of ‘behaviour’ and ‘results, but we have recently sought to gather data from participants and other stakeholders to consider these impacts more systematically. In this preliminary investigation, three cohorts of informants were identified – firstly, students who had participated in at least one residential course in the past three years, secondly students who had actively contributed as course facilitators and finally, academics in two UCL departments which had engaged with CALT’s skills provision for postgraduates, both credit-bearing and not, over a number of years. In addition to the programme and ancillary courses provided via the Graduate School’s Skills Development Programme, CALT provides credit-bearing modules within its MA programme, taken by research masters and doctoral programme candidates from a range of disciplines across UCL. A compilation of participant response after a year-long engagement with the ‘Professional Development in Practice’ module is also appended.

Two kinds of thresholds for achieving long-term impact are apparent from these initial investigations. Firstly, the deeper impact arising from a continuity of engagement over time, particularly in undertaking responsible roles within the facilitator development programme; this is invariably at the initiative of the individual participant, irrespective of departmental support. Secondly, the deeper impact arising from a consistency between the guidance given in the programme and the expectations evident in departmental practice, providing a context in which the learning is made meaningful and embedded. The following pages offer some elaboration of these apparent trends.

Are there any pre-requisites for engagement, e.g. levels of skill, years of experience, essential pre-activities?

How many participate in each 'activity'?

How do you monitor effectiveness?

Who do you seek feedback from?

Do you have benchmarks?

Evaluation evidence:

The evidence of impact related to the “Rugby Team Impact Framework” is based on the results of the summary reports of the end of course evaluation forms for academic years 2007/08 and 2008/09, as well as the results of the online questionnaire. The conclusions of the interviews conducted for the purpose of the documentary video, have not yet been included, neither have the reflections of academic staff.

Level 0 – Foundation
The input provided by participants at the end of each course helps in improving its overall design. Due to the residential nature of the course, some of the comments relate to the site facilities and provisions and are used to improve the overall experience of the course. Examples of the comments that influenced the organisation of the course include restructuring the timeline of the first day to accommodate time for social networking, changes in skill studios content or involving team facilitators in creation of coaching network.

Level 1 - Reaction
Over the years the course has received overwhelmingly positive responses from attending students. When judging their initial reaction in the end of course evaluation form, participants almost universally declared their willingness to recommend the course to other postgraduates (on average 84% - yes, definitely; 15% - yes, probably, 2% - unsure) and rated the course very highly, with the average of 94% of participants awarding one of three top ratings (from 0 – poor to 9 - excellent).

Level 2 - Learning
The initial assessment of the realisation of course outcomes by participants indicates significant learning, with the achievement of course objectives rated by 89% of respondents as ‘fully’ or ‘mostly’ realised (Figure 1). All participants had made progress in the realisation of their personal objectives, set at the beginning of the course (fully or mostly achieved for the average of over 77% of participants). Participants also identified unexpected outcomes of the course, achieved fully or mostly for the average of 87% of them

Level 3 – Behaviour
Items in the online questionnaire were constructed to provide some insight into the impact of the programme over a longer time scale and at deeper levels. The interval between course participation and responding to the questionnaire varied from four months (in case of participants attending November 2009 course) to as much as ten years after completing the course (two of the facilitator cohort). Free text questions were designed to obtain qualitative indications of behavioural changes.
Among many examples participants had mentioned improved understanding of human nature, teamwork and team dynamics, which in turn improved their conflict resolution skills and accommodation of others' working styles. Many reported having gained better understanding of their personality contributed to their working and communication styles, and significant improvement in overall confidence and in creativity. A major theme in the benefits reported was improved awareness of individual skill strengths and areas needing further development, consistent with the design of the course. Many had used these skills directly in job interviews, and considered them a factor in their successful organisation of research projects and in collegial engagement.

Level 4 – Results
Respondents were asked to estimate the influence of the course on the completion of their degree (Figure 4), as a shorter term measure of impact. Very few participants were unable to identify the specific effects of the course on their work as students, with the majority reporting at least two areas of improvement. As may be seen from the chart, the cohort who had participated in the programme over a sustained period (facilitators) indicated significantly greater benefit accruing within their degree registration period to successful completion. A possibly related observation is the reportedly greater impact of course participation over a longer timescale, beyond the period of being a student and taking part in the programme. Among free text evidence cited were reported successes in job interviews, degree completion and management of a variety of projects.