The medium- to long-term evaluation of the initial and continuous training courses organised by the Italian School for the Judiciary
In March 2022, a collaboration contract was signed between the Italian School for the Judiciary (SSM) and the Bologna branch of the Institute of Legal Informatics and Judicial Systems (IGSG-BO) of the National Research Council of Italy (CNR) for the collection, data processing and statistical analysis of the medium- to long-term evaluations of the initial training courses and permanent training courses organised by the School itself.
In agreement with the School, two main activities were carried out: 1) the preparation and subsequent statistical processing of the data obtained through the online administration of a questionnaire for the mid- and long-term evaluation of the School’s training activities carried out within the framework of the courses for ordinary magistrates on traineeship (MOT), 2) the preparation and implementation of three focus groups for the mid-term evaluation of some permanent training courses identified by the School itself.
The objectives of the activity carried out were mainly two: a) verifying whether the methods used allow for an effective ex-post evaluation of the courses according to the Kirkpatrick methodology (levels 2 and 3) chosen by the European Judicial Training Network (EJTN) to evaluate the training activities carried out by the European Judicial Training Schools;4 b) collecting useful information to improve the School’s training offer some time after participation in the courses.
The methodology developed by Kirkpatrick comprises four levels of evaluation. Level 1 concerns the immediate evaluation of the course and the lecturers (so-called ‘reaction’, react). The School, as is known, already carries out this evaluation through the completion of a questionnaire by the participants at the end of each course. Level 2 of the evaluation aims to measure whether the course enabled the acquisition of new competences, competences or attitudes (so-called ‘learning’). The instruments suggested by the Kirkpatrick methodology to evaluate this type of learning are the questionnaire, self or group evaluation, focus groups, and individual interviews, evaluation by a superior, observation of activities before and after the course. Level 3 would like to measure whether and what kind of change took place in the workplace after the course (so-called ‘change’). In this case, the evaluation instruments are similar to those of the previous level to which discussion and evaluation among colleagues (peer review or intervision) are added.5 Level 4 has the ambition to measure what the overall results were after the training activity (‘results’). The applicable instruments are again the questionnaire, evaluation among colleagues, action plans, evaluation by external or internal experts, and a user satisfaction questionnaire.
Bearing in mind the objectives, resources and timeframe of the research activity, it was decided together with the School to proceed with an online questionnaire for the ex-post evaluation of the initial training courses for trainee ordinary magistrates, whereas the focus group method was preferred for the evaluation of the permanent training courses.
The choice of the questionnaire seemed most appropriate for the evaluation of initial training courses because they are rather standardised courses, with training proposals that have been quite similar over the years, aimed at new magistrates and therefore at rather large groups with homogeneous characteristics. For these reasons, the questionnaire seemed to be the most suitable tool for the ex-post evaluation of initial training. The analysis of the data collected with the questionnaire was then the basis for a subsequent specific focus group with some MOTs, an activity that further enriched the indications provided by the questionnaire.
For the ex-post evaluation of permanent training courses, the focus group instrument was preferred due to the heterogeneity of the courses and participants. The courses usually see the participation of magistrates with a variety of experiences, with a significant number of participants but in any case, lower than those of the MOTs (never more than one hundred per course), and above all a variety which, in order to attempt to be evaluated, requires calibrating the collection of information on the individual course with a more qualitative and, to some extent, in-depth analysis.
This final report consists of three parts. An initial part summarises the main results of the activities carried out, also dwelling on the methodological aspects aimed at verifying the usefulness and feasibility of the Kirkpatrick method suggested by the EJTN. The second part is the research report on the data analysis of the ex-post evaluation questionnaire of initial training. The third part collects the information gained from the focus groups for the evaluation of continuing training courses.
