Authors: CSIL: Julie PELLEGRIN, Louis COLNOT, with support from Matteo PEDRALLI Country experts: University of Warsaw Diana IONESCU (RO), Tomasz KUPIEC (PL) Agnieszka OLECHNICKA (PL) CSIL: Matteo PEDRALLI (IT) ESTEP: Neringa Viršilienė (LT) Scientific Advisers: Free University of Brussels-VUB: Nicola Francesco Dotti and University of Milan: Massimo Florio
Objectives and background
This study provides members of the Committee on Regional Development (REGI) with an in-depth analysis on the role of evaluation in Cohesion Policy. This takes place at a critical juncture when the Cohesion Policy’s post-2020 budget is likely to be reduced, and the regulations defining the basis of the policy are still being negotiated.
The specific objective of this study is to assess the effectiveness and efficiency of the framework providing for the evaluation of Cohesion Policy at the level of the European Union (EU) and Member States. For the 2014-2020 programming period, this framework comprises rules and procedures enshrined in the Common Provision Regulation (CPR) and fund-specific regulations, as well as a series of soft support instruments and guidance documents. This framework is expected to fulfil different functions and objectives, including the consolidation of a result-oriented approach to Cohesion Policy, the development of evidence-based policy learning, and the provision of relevant evidence on the impact of Cohesion Policy for reasons of accountability (i.e. justifying the spending of taxpayers’ money).
To address these issues, this study relies on a methodological approach combining different tools, including a literature review, interviews with relevant stakeholders, and case studies conducted in six Member States (France, Ireland, Italy, Lithuania, Poland and Romania).
Over the years the European Commission (EC) has made considerable effort to deploy an ambitious evaluation framework for Cohesion Policy (CP), the most evaluated EU policy. Its building blocks are the definition of three types of evaluations (ex-ante and ongoing evaluation under the responsibility of Member States, and ex-post evaluations mostly realised by the EC) and institutional arrangements in the Member States structured around a Managing Authority (MA) and Monitoring Committee.
During the 2014-2020 programming period, all three types of evaluations have been mandatory to ensure their completion at different stages of the policy process, across Member States. Mandatory Evaluation Plans encourage MAs to adopt a long-term strategic approach to CP evaluations. Monitoring indicators are closely linked to the programme’s intervention logic and reinforce result orientation. Specific indicators are expected to give evidence on implementation progress necessary for MAs to introduce adjustments, and common indicators show CP achievements at the EU level. Substantial efforts have been made by the EC to support Member States and MAs by providing guidance documents and soft support (e.g. promoting impact evaluation approaches).
How this framework is actually implemented in practice at the EU and Member State levels shows a mixed picture. The CP evaluation framework facilitates an increasingly accurate analysis of CP achievements at the EU level, with improved indicators and steady progress in the realisation of ex-post evaluations by the EC. Some limitations remain concerning the possibility to aggregate indicators at EU level while mainstreaming evaluation findings in policy making needs further consolidation at EU level.
The Member State level presents more difficulties. Despite progress, and evidence that an evaluation-based learning process is taking root in some Member States, the quality of these evaluations remains mediocre and it is difficult for evaluation findings to feed into policy process at this level. The reasons for this include some requirements, which are too-strict (e.g. in terms of timing or coverage of evaluations), a lack of MA resource and capacity, and a weak ‘evaluation culture’ in some Member States. As a result, CP evaluations are sometimes seen as an exercise generating more ‘administrative burden’ than benefit and arrangements are often set formally to merely comply with requirements.
In this context, the Commission’s post-2020 proposals aim at simplifying rules by reducing mandatory provisions in the CPR and fund-specific regulations, and limiting written guidance. For example, ex-ante evaluations are no longer required, programme-specific indicators are not mandatory and the dedicated approval procedures for major projects is removed. Also, the legislative proposals do not refer to the obligation for MAs to ensure evaluation capacity and for the EC to provide guidance. Some new obligations are introduced, such as the need for MAs to use evaluation criteria as defined by the Better Regulations framework, and more frequent data reporting instead of compiling Annual Implementation Reports.
The rationale underlying the Commission’s proposal to simplify the evaluation framework is to grant more flexibility to Member States in order to improve their ‘ownership’ and commitment. However, as shown by past programming periods, the risk is that without a legal basis, some Member States and regions (especially those most behind) will not autonomously implement arrangements at the core of the evaluation practices that the EC has contributed to consolidate over the years. At best, a fragmented system of country-specific practices and approaches would emerge. This could undermine the considerable effort made so far in raising quality standards and harmonising practices and approaches.
It is therefore necessary to strike a balance between simple, effective regulatory provisions that show what needs to be done, and giving MAs the flexibility to organise and implement requirements in accordance with local specificities. This should form the basis for a sound partnership between the EC and Member States. The European Parliament has an important role. It should adopt a proactive approach to law-making as this is an important pre-requisite for effective evaluation. It should take stock of evaluation findings on Cohesion Policy, among other evidence, to raise the political dimension of the latter, and it should contribute to an evaluation culture in Member States and regions.
Based on the findings from the study, two types of recommendations are envisaged. First, strategic recommendations focusing on structural and longer-term issues suggest to:
- Make explicit the purpose of CP evaluations at the Member State level regarding both accountability and policy-learning objectives. This has implications for how evaluations should be conducted, by whom and when – overall, for their conditions of effectiveness;
- Promote a more participatory approach to overcome the lack of commitment of Member States. The EC, Member States and regions could co-design the evaluation framework;
Second, a series of operational recommendations can be addressed to the main stakeholders.
- Co-legislators (the European Parliament and the Council) should consider revising some provisions of the future CPR and fund-specific regulations during the ongoing negotiations. For example:
- Engage the European Parliament along the lines in the paragraph above and detailed below.
- Ex-ante evaluations should be mandatory and could be done by authorities other than MAs to challenge the mainstream vision and make them more relevant;
- Managing Authorities should have flexibility in deciding the coverage (programme or axis level, thematic) of impact evaluations;
- The timelines proposed should be indicative, or dependent on absorption of funds;
- Requirements concerning indicators should be included, such as the need to design programme-specific indicators and to ensure that common indicators can be aggregated at EU level for comparison purposes;
- It should be clarified that the five evaluation criteria of the Better Regulations framework are a basis for defining specific evaluation questions adapted to the programme being evaluated;
- The dedicated selection and assessment procedure for major projects should be reintroduced to preserve a harmonised approach to infrastructure appraisal at the EU level;
- Building capacity in Member States and the obligation for the EC to provide support should be enshrined in the regulation to help align requirements with Member States’ ability to fulfil them.
- At the Member State level, the improvement of evaluation capacity and culture should be pursued and supported by adequate resources. Professionalism of evaluation teams should be an explicit objective. Discussions of evaluation findings should be promoted, including in political debates. The possibility of substituting or flanking the MA with a competent alternative body when the MA cannot guarantee a sufficient level of capacity should be considered.
- The European Commission should build upon its progress so far and keep consolidating evaluation capability in Member States and regions. It should strive to strike a balance between prescription and flexibility and continue supporting the regions most in need, in the most effective way.