Coherence Scoring
The scoring mainly provides a visual aid to convey synthesized information based on more extensive qualitative analyses. The approach to scoring in the CrossGov Policy Coherence Evaluation Framework is based on Nilsson et al.’s policy coherence analysis methodology (Nilsson, Griggs, and Visbeck 2016). It combines elements of coherence scoring with qualitative analysis of data from policy documents, expert input and case studies.
The scoring should be based on the qualitative answers to the guiding questions addressed in the respective assessments.
In order to ensure robust scoring results, internal calibration of scoring approaches between the involved researchers is necessary. This requires an open discussion before the scoring exercise to agree on the approach to be used to decide on the scores. Scoring could be based on a literal reading of the policy documents, interpretations of the text (by oneself or as provided by jurisprudence), the viewpoints of key responsible governmental authorities, or affected stakeholders, and so forth. Divergent scoring approaches can affect the robustness of the results. For that reason, calibration ex ante and throughout the process is important.
See below the criteria to guide the scoring of the coherence attributes (explanatory variables were not scored in CrossGov).
Negative/ - 1
Neutral/0
Positive/1
Instructions
Assessing policy objectives against objectives
Objectives are not aligned and contradict or impede each other.
Achieving the objectives of policy A would make it difficult/impossible to achieve the objectives of policy B.
No explicit reference to objectives from other policies.
No explicit reference of contribution to the EGD objectives.
Spatial and temporal scales of the objectives are mismatching across policies.
Objectives are similar, overlapping, mutually reinforcing, or inextricably linked
Achieving the objectives of policy A would be complementary to/support achieving the objectives of policy B.
Objectives from other policies are explicitly referenced.
Contribution to EGD objectives is specifically referenced.
Spatial and temporal scales of the objectives are aligned across policies.
Assessing policy instruments against instruments
The instruments address issues in isolation and have conflicting implementation mechanisms
Compliance with the implementation mechanisms of policy A makes it difficult/impossible to comply with implementation mechanisms of policy B.
Spatial and temporal scales of the policy instruments are mismatching across policies.
There is no cross-fertilization of knowledge, and knowledge and data is fragmented despite shared knowledge needs across policies.
There are no cross-sectoral coordination/ collaboration mechanisms to resolve conflicts between policies.
Coordination/collaboration is hampered by power imbalances or lack of communication, infrastructure, resources, political will, etc.
The instruments consider issues in a joined-up way and have shared implementation mechanisms
Policy A and B share implementation mechanisms (e.g. integrated licensing systems, shared monitoring frameworks, common indicators).
Spatial and temporal scales of the policy instruments are aligned across policies.
The instruments use, and contribute to, knowledge and data from other policies (cross-fertilization of knowledge).
Cross-sectoral coordination/collaboration mechanisms are in place to avoid and resolve conflicts (e.g. inter-organizational consultations/committees, joint stakeholder groups, joint decision-making bodies).
Coordination/collaboration is facilitated by equitable and inclusive stakeholder engagement, good communication, appropriate infrastructure and resources, strong political will, etc.
Assessing policy instruments against policy objectives and/or EGD objectives
References
- Nilsson, Måns, Dave Griggs, and Martin Visbeck. 2016. “Policy: Map the Interactions between Sustainable Development Goals.” Nature 534 (7607): 320–22. https://doi.org/10.1038/534320a.