2. Terminology 5
3. Monitoring process 8
4. Evaluation process 10
5. Baseline and impact measurement 12
6. Indicators 13
7. Project monitoring using the logical framework 15
8. Data collection 17
9. Monitoring and evaluation tools 32
10. Data analysis 39
11. Information 42
ANNEXES: Annex 1: KPI Indicators Sheet 45
Annex 2: Questionnaire on “impacts of Achieved Results on members” 67
The World Meteorological Organization (WMO) Monitoring and Evaluation (M&E) Manual sets out the details and provides guidance on the monitoring system to establish the progress in achieving results defined in the WMO Strategic Plan.
A fully functioning M&E system provides a continuous flow of information that is useful, both internally and externally, to measure the progress being made towards the achievement of expected results, which reflect the desired outcomes to be achieved by WMO as a whole (that is, WMO Members, WMO constituent bodies, WMO Joint Programmes and the Secretariat). The associated key performance indicators (KPIs) are the agreed measurements that will enable WMO, as an organization, to assess the achievement of the expected results.
Achieving expected results is a complex process involving coordinated efforts of the main WMO components (Members, WMO constituent bodies and Secretariat), and partners, such as IOC/UNESCO, UNEP, UNDP and international funding institutions.
Internally, within the WMO Secretariat, M&E information is a crucial management tool towards achieving results and meeting specific targets. Information on progress, problems and performance are all central to organizational success. Information is also important for organizational learning, improvement and to build stronger external relations, as well as to identifying other worthwhile activities to allocate scarce resources.
M&E provides critical information and empowers policymakers to make better, more informed decisions. At the same time, providing such information may ensure consistent focus on efforts to achieve objectives of an organization. Instituting M&E systems that highlight outcomes—both successes and failures—and that provide greater transparency and accountability may be challenging.
Objectives of the WMO M&E system The objectives of the WMO M&E system are:
To enable senior management and programme managers of the Secretariat, the WMO constituent bodies and the scientific steering committees of WMO joint programmes to track the progress of the Organization’s activities, enhance performance and impact, and identify areas where corrective action is required with respect to activities and deliverables.
To enable the Secretariat to demonstrate organizational effectiveness, efficiency, accountability and transparency externally and when reporting to WMO governing bodies.
To facilitate dialogue and discussions at meetings of WMO constituent bodies, with partner organizations and donors, as well as other stakeholders.
The Members are the main beneficiary to ensure that the WMO is delivering the agreed expected results in a cost efficient and effective manner. Internally, within the Secretariat, the benefits are applied to all departments to improve their delivery and ensure that the activities directly lead to the achievement deliverables/outputs associated with the outcomes of an expected result.
The Congress and Members need evidence to show progress against outcomes and to see that the funds are being used efficiently and effectively. The Executive Management and Heads of Departments at the Secretariat need to have regular progress reports to measure progress against accomplished deliverables and budget expenditure.
Strategic principles The WMO M&E System is based on performance measures in the WMO Strategic Plan and Operating Plan, and takes into consideration the following principles. It has the following essential properties:
An M&E system connecting the outcomes achieved to the use of resources, implementation of activities and production of deliverables/outputs.
A bottom-up, flexible approach to ensure relevance, credibility and accuracy in the system, with uniform formats for reporting purposes.
An M&E system that will evolve, over time, to allow experience and lessons learned to be captured.
A results-based culture that should underpin the M&E System. Learning is consistent with a focus on programme improvement and on identifying new approaches that may be needed to address recurring or new needs.
Quality assurance of performance data is important, while acknowledging that information may never be perfect.
An integrated approach to monitoring and evaluation that is key to recognizing that an effective performance management system requires both.
Consistent with the WMO Quality Management Framework initiative and the ISO 9004:2000 quality management systems; guidelines for performance improvements.
A priority, for cost-efficient solutions for the implementation and operation of the WMO M&E System, in line with Executive Council (EC) requests.
Performance measures The WMO M&E system is based on the following performance measures:
Performance metrics as stipulated in the WMO Strategic Plan (key performance indicators (KPIs) and expected results) and in the WMO Operating Plan (key performance targets (KPTs) and deliverables).
Performance baselines, quantitative or qualitative, that provides data at the beginning of, or just prior to, the monitoring period and is used as a starting point to monitor future performances. Baselines are the first critical measurement of KPIs and KPTs.
Performance targets which reflect the desired achievements (status of outcomes) at the end of the monitoring period. Organizational performance during a given period is measured on the status of the KPIs and KPTs, as compared to the baseline situation (that is, targets versus baselines).
The expected results reflect the desired outcomes to be achieved by WMO as a whole (that is, WMO Members, WMO constituent bodies, WMO joint programmes and the Secretariat) and partners.
Why measure results?
If you do not measure results, you cannot tell success from failure.
If you cannot see success, you cannot reward it.
If you cannot reward success, you are probably rewarding failure.
If you cannot see success, you cannot learn from it.
If you cannot recognize failure, you cannot correct it.
If you can demonstrate results, you can win public support.
Results-based management (RBM) focuses on improving management effectiveness and accountability to achieve the intended results of WMO. It does this by addressing several key questions:
What activities have taken place?
What deliverables/outputs have been realised from the implemented activities?
What outcomes have been realised from the accomplished deliverables/outputs?
What has been the impacts (effects) and result of these outcomes?
How have we contributed to the strategic results?
How can we prove all of this?
A number of terms need to be understood and recognized and detailed below.
Activity Is an action, or a group of actions, implemented to generate products or services. The implementation of one or more activities leads to the production of a deliverable/output. Activities stipulated in the WMO Operating Plan are based on the contributions of the technical commissions, regional associations and Secretariat. Additional activities that contribute to the achievement of expected results are related to the WMO joint programmes and Members.
Baseline A set of quantitative and qualitative data gathered early in the life of a programme for the purpose of establishing a benchmark against which to measure change.
A lower level, expected result (Secretariat specific) that is arrived at through an accumulation of activities, i.e., is a tangible outcome. Deliverables contribute to the achievement of a key outcome or programme area. The term “deliverable” and “output” are interchangeable at WMO.
Inputs Inputs are the financial, human, and material resources from the Members, TCs, RAs, WMO Secretariat, WMO joint programmes and partners used to implement the Strategic Plan to achieve expected results.
Expected result (ER)
An expected result is a top level statement that predicts a high level outcome (change in Members’ or societal conditions) to be achieved in the long term by WMO as a whole (the Secretariat, technical commissions, regional associations and Members).
Evaluation Evaluation is an intermittent process that aims to provide an objective assessment of an ongoing or completed intervention. It is likely to include an assessment of its design, implementation and results in order to determine its relevance, fulfilment of objectives, efficiency and effectiveness, as well as its overall impact and sustainability.
Evaluation addresses the strategic questions, ‘What?’(impact and sustainability) and ‘Why?’ (relevancy). Here, the analysis is getting deeper and seeks actual cause–results relationships and eventual implications of the observed situations. It seeks the ‘big picture’ conclusions and, for WMO, means completion of the eight strategic results.
Global Societal Needs (GSN) Needs identified in the global society which WMO, through its mission, seeks to address and which forms the basis for WMO’s strategic direction.
Impact In its broadest sense, impact assessment is the process of identifying the anticipated or actual impacts of an intervention, which the intervention was designed to affect, or affected inadvertently.
Key Outcome (KO)
Key outcomes are the likely or achieved short-term and medium-term effects of accomplished deliverables/outputs related to programme areas that define the parameters for the unique contribution by WMO to in the progress addressing the achieve expected results. There are several outcomes for each expected result.
Key Performance Indicator (KPI) A quantitative or qualitative metric used to measure the achievement of, or the progress made towards achieving, an expected result. A key performance indicator is specific, measurable, achievable, reliable and time-bound.
Monitoring Monitoring is a continuous, internal process that uses a systematic data collection, collation and analysis process on specified results-based indicators (measures). It provides management and key WMO stakeholders with an indication of management progress toward achieving agreed outcomes, as well as the efficient use of allocated funds.
Regular monitoring enables identifying actual or potential problems as early as possible in order to facilitate timely adjustments in implementation. Monitoring, in a results-based context, is also concerned with assessing the status of performance; that is, whether results have been achieved.
Monitoring is focused on daily management issues. The typical questions are: “How many?” "When?” “How?” “For how much?” By monitoring, we try to assess whether activities are implemented effectively and efficiently.
Monitoring is expected to generate useful information and identify where bottlenecks may be. How are we doing towards our expected results? Are we being efficient in terms of resources and budget?
Monitoring refers to a predefined programme’s strategic framework that guides implementation. It is expected to generate timely information on operational efficiency and effectiveness. To fulfil these needs, monitoring utilizes ‘rapid assessment methods’, which provide fast feedback and are not very expensive.
WMO Result-based Budget (RBB)
WMO Results-based budget identifies resources derived from assessed contributions of Members, as well as from voluntary contributions to implement programme activities and projects in the WMO Operating Plan. It is approved once every four years by Congress. A schematic representation of the linkages among the WMO Strategic Plan, Operating Plan and RBB as well as monitoring and evaluation is presented below.
Strategic Thrusts (ST) Are broad indications of strategic directions to address the global societal needs (GSN) to achieve expected results.
Target The desired level of performance (milestones) to be achieved during a financial period, i.e. (performance=baseline + x, where ‘x’ is the change experienced).
WMO Operating Plan WMO Operating Plan provides details on key outcomes, deliverables and programme activities and projects contributing to the achievement of expected results, which are used to guide resource estimates and allocations in the results-based budget.
WMO Strategic Plan WMO Strategic Plan indicates the unique strategic contribution of WMO to the attainment of the global societal needs. It is composed of the following elements: global societal needs, strategic thrusts, a series of expected results and strategic priorities.
The first step in monitoring process is to define the results that need to be measured and the metrics for measuring them. The results to be measured and performance indicators to measure them are defined in the WMO Strategic Plan and Operating Plan. Those responsible for collection will select methods and tools for data gathering (What, Where, When, Who, Why?) and methods for checking data, recording, collating and analysis (What, When, Where, Who, Why?). The information collected is then e analysed and results communicated.
Every six months, progress against deliverables/outputs that contribute to the achievement of the key outcomes associated with expected results are monitored to ensure timeliness, quality and cost efficiency. The system is based on planned, costed activities having been set for each ‘deliverable/output.
Monitoring of these deliverables is carried out within departments and the data is used by the Strategic Planning Office to compile a progress report. The system allows comprehensive progress reports to be produced and progress to be shown against deliverables/outputs and the associated key outcomes and expected results.
Monitoring functions in a reverse sense of the strategic and operational planning process; that is, as a ‘bottom-up’ process. A schematic representation of the monitoring process is provided below:
For monitoring purposes, the basic data collection is performed at the level of the individual activities. These activities are linked to the deliverables, the achievement of which is measured by the KPTs. Activity codes correlate to budget codes to allow for uniform reporting. A similar process is used to link the deliverables to expected results. This time measured by the KPIs.
The second focus of monitoring is on the performance indicators (that is, the KPTs and the KPIs) to ensure that programme implementation is on target towards achieving results. This set of data will allow programme managers to determine whether:
A deliverable/output has been achieved as a result of the set of activities implemented.
Key outcomes and expected results have been achieved as a result of a set of deliverables.
Prior to each financial period, the WMO constituent bodies and the Secretariat departments will establish their respective monitoring approach to best suit and support the implementation of their work programme. This approach would be outlined in the respective contributions to the WMO Operating Plan and would address two monitoring levels.
The first level monitoring, to be conducted on a six monthly basis, is concerned with the collection of data and analysis of the following elements:
The tracking of the use of resources (post and non-post) within the allotted budget allocations (this concern.
Timeliness, quality and cost-efficiency of activities and outputs (meetings, capacity building events, studies completed, persons trained, technical guidance and instructions).
The second level monitoring, to be conducted annually, is concerned with the collection of data and analysis of the following elements:
Progress made or achievement of deliverables (measured on the KPTs).
Progress made towards achieving expected results, as compared to the baseline (measured on the KPIs).
Institutional monitoring at WMO is concerned with the internal monitoring of financial, physical and organizational issues. Financial monitoring tracks inputs and costs by activity within predefined categories of expenditure. Physical monitoring tracks the distribution and delivery of project activities and outputs or interventions.
Organizational monitoring tracks sustainability, institutional development and capacity building in the project and direct partners. Results monitoring tracks effects and progress towards completion of the expected results.
These monitoring activities vary in terms of where the data is collected from (that is, the source), the frequency of collection and the methods used for gathering and analysing the data.
Inputs are assessed by monitoring financial information. Monitoring input data helps keep the WMO management informed of the degree of financial efficiency with which the organization is operating. Inputs include physical and human resources (the means) and financial resources (the costs). These data are managed according to specific expenditure categories and are reported in regular financial reports.
Physical monitoring is carried out to assess progress in the delivery of outputs associated with implemented activities with regard to the effects on the target groups and timeliness. The sources of information for physical monitoring include various WMO records. Information is sought that not only informs on the achieved targets but also to explains the progress. Only by knowing why something is happening or why not, do we have a basis for deciding what corrective action is needed.
Tracking information related to the objective hierarchy will only keep you up-to-date on what you intend to achieve. Seek out unintended positive and negative impacts in order to take any corrective action that might be necessary. Last but not least, stick to the ‘less-is-more’ principle. Regularly revise your list of information needs to filter out the information that is not critical to manage possible impact.
Details of the KPI for the period 2012-2015 are provided in the Annex 1, where the data collection process is presented.
The WMO M&E system comprises two types of evaluation:
Self-evaluations (to be conducted by the WMO constituent bodies and the Secretariat).
Independent evaluations (to be undertaken by expert groups from the technical commissions, particular groups of the regional associations, or individual external or internal evaluation/technical experts, or by internal and external oversight).
Both self-evaluations and independent evaluations will be conducted periodically. In case concerns arise during implementation of the WMO Operating Plan, a more in-depth review of programme implementation to support decision-making would be warranted. An evaluation could be triggered when:
There is a divergence between planned and actual performance.
The links between implementation (activities and outputs/ deliverables) and expected results is unclear.
Making resource allocation decisions.
There is conflicting evidence of results.
There is a need to validate results information or provide an impartial assessment of programme performance (independent evaluation).
At the beginning of each financial period the WMO constituent bodies and the Secretariat departments will establish their respective evaluation approach as part of their contributions to the WMO Operating Plan. They will include a description of self-evaluations to be conducted on a biennial basis. Self-evaluations should address issues of relevance, effectiveness, efficiency and sustainability of the achievement of deliverables and expected results. In particular, they should look into:
The achievement of deliverables and expected results compared to baselines.
The satisfaction of Members.
External factors that have affected implementation.
Challenges encountered during implementation.
Corrective actions taken and eventual adjustment of strategies.
What worked well and why.
Independent evaluations will be carried out upon request by the EC and Congress, possibly based on recommendations by the Secretary-General and the WMO constituent bodies. Such evaluations could cover, for example, a project, programme, specific work areas of the technical commissions, or work programmes in particular regions.
Any decisions to modify expected results from one financial period to the next should be preceded by an independent evaluation to support decision-making.
To produce objective and exact information, evaluation uses more scrupulous research methodologies, such as representative surveys and comprehensive quantitative analyses. Reviews of WMO outputs, outcomes and impact are conducted on the back of well-recognized trends in the surrounding environment.
The achievement of expected results is the ultimate benchmark for organizational performance. Therefore, the responsibility for the assessment process of whether an expected result has been achieved will ultimately reside with the EC and Congress.
Baseline and Impact measurement
The baselines for the KPIs for the period 2012-2015 will be established from the results of the survey of the impacts of achieved results on Members conducted in February 2012. They will ascertain the base level for each KPI and the associated key outcome and expected result. Other similar surveys will be conducted in the middle (2013) and end (2015) of a financial period to measure progress in achieving expected results. The data gathered at the end of a financial period shall be compared to the situation at the beginning of the financial period to establish the progress in achieving results. The data at the end of a financial period shall also serve as the baseline for the following financial period.
The main tool for gathering data on impacts is the questionnaire on ‘Impacts of Achieved Results on Members’ that is completed by Members twice in a financial period (in the middle and at the end). The questionnaire is presented in Annex 2. It provides a measurement of KPIs as perceived by Members.
Types of data collected are qualitative and quantitative. The quantitative data derived from formal questionnaire surveys measures differences in variables that can be counted. Qualitative data is descriptive and measures beneficiary perceptions and attitudes.
It is common to use both types of surveys. Qualitative baselines add depth to quantitative baselines, which in turn add statistical rigour to support the results of qualitative baselines.