TERMS OF REFERENCE
Development of a Monitoring and Evaluation Framework
for the Enabling Income Generation for the Poor (EnInGeP) Project
GENERAL INFORMATION
Job title: Monitoring and Evaluation Expert
Project title: Enabling Income Generation for the Poor (EnInGeP) Project
Donor: Austrian Development Agency
Place(s) of assignment: Kosovo
Assignment duration: TBD, on the basis of the offer and agreement with the Project Manager
Remuneration type: Daily fee/rate
ABOUT CARE INTERNATIONAL BALKANS
CARE’s work in the Balkans started in 1993, when it provided humanitarian support to people affected by war. In the late 90-ties, CARE shifted its focus in the region from humanitarian post-war assistance and rehabilitation to socio-economic development, engaging in interventions directed at conflict prevention and peace-building, sustainable livelihoods, gender equality and the prevention of gender-based violence.
CARE International in the Balkans places the quality of its program at the centre of its mission and has developed a regional strategy that encompasses two main program directions: Gender Equality and Social and Economic Inclusion. CARE’s Gender Equality Program aims to empower women vulnerable to violence, discrimination and poverty, to reach better life opportunities and social justice and to work on prevention of peer violence and building of tolerance among young men in the Western Balkans. The goal of the Social and Economic Inclusion Program is to strengthen capacity and create opportunities for the marginalized, socially excluded and poor to integrate into society and access rights. CARE’s engagement and contribution lies in strengthening sustainability of key regional, national and/ or local civil society organizations and networks promoting equality and diversity, in the context of social inclusion and non-violence.
PROJECT BACKGROUND
While Kosovo is grappling to achieve, inclusive, and sustainable development, an array of structural market constraints across a range of developmental domains continue to perpetuate exclusionary practices. The poor business environment, inadequate access to finance, low Foreign Direct Investments (FDI) inflows, inequitable gender norms, etc., are just some of the structural hindrances that undermine Kosovo’s competitiveness.
Access to finance prevails as a critical market constraint for the expansion of companies in Kosovo. It is challenging to secure funds required for business development and growth. There is a lack of publicly available information on MSMEs’ financials, making it more likely for banks to avoid vulnerable micro and small companies. Women in Kosovo continue to face difficulties in securing access to bank financing, due to owning barely any property. Difficulties relating to access to finance stem also from the high levels of financial illiteracy in Kosovo. The attempts of the Central Bank of Kosovo (CBK) and other actors to enhance financial literacy, have largely stagnated. On the other hand, the Kosovo Credit Guarantee Fund (KCGF) has contributed substantially toward facilitating the access to finance for Kosovo MSMEs. This mechanism was established with the support of donors including ADA providing the credit guarantees for MSMEs, resulting in supporting local production and creating new jobs.
One of the sectors with the highest potential for spurring more inclusive growth through the export of services is the ICT sector. The severe skill shortage in the ICT sector in Europe is expected to translate into amplified demand for ICT outsourcing. Based on Kosovo IT Barometer, 91.18% of IT companies export their services, whereas only 5.8% of companies work in the domestic market.
Expanding women’s access to ICT jobs is regarded as not only advancing economic opportunities for women and their communities, but also addressing the mismatch between the supply and demand for jobs in emerging countries. In addition, the more sections of society (women as well as other marginalised groups) are involved in the design and development of ICT products and services, the more these will speak to a larger consumer base. Women’s participation in the ICT sector therefore not only triggers economic and social empowerment for women which benefits their families and communities, but it enables sectoral growth and the related and potentially significant financial rewards. Leveraging the ingenuity of the entire population could trigger economic growth by increasing productivity across sectors. In Kosovo, the participation rate of women in ICT is 25% holding varying positions, from junior level to key leadership. A larger women’s participation, is hindered by an array of factors, including:
- norms discouraging girls and women from pursuing the education and training required to join the ICT workforce.
The Business Development Service (BDS) industry in Kosovo, despite noticeable improvements, continues to suffer from the lack of specialized consultants and standardization. Consulting in Kosovo is accompanied by prejudices, which, to some extent, can be linked to difficulties in the evaluation of consulting services, and the fact that the profession of the consultant is not regulated. Based on the research conducted within ACCESS project the use and availability of BDS is shown to be significantly higher in urban as compared to rural areas, and much more present among members of the Albanian community, as opposed to those with an ethnic minority background. As far as gender differences in BDS use are concerned, a survey among BDS providers conducted during ACCESS, showed that women entrepreneurs use BDS to a similar extent as men.
PROJECT IMPACT
To contribute to a sustainable, resilient, and inclusive private sector development that enables inclusive job creation and income generation for the poor and vulnerable.
PROJECT OUTCOMES
The interventions proposed will achieve the following outcomes:
PURPOSE OF ASSINGMENT
CARE is looking to engage a Consultant to develop a comprehensive monitoring and evaluation (M&E) framework, implementation plan clearly outlining distinct staff and partner roles and responsibilities, schedules for M&E data collection, and other relevant M&E tools all in accordance with CARE guidelines for monitoring, evaluation, accountability and learning (MEAL), and DCED Standard to be used for assessing and understanding the progress of the project impact, outputs, and outcomes. The Consultant will provide the roadmap to monitor and evaluate the project’s contribution to achieving the desired project impact, outputs and outcomes, with SMART, well defined and CARE aligned indicators.
Implementation progress, project reach, and (intended and unintended) effects on end users will be assessed as well as success in building capacity and influencing policies.
SCOPE OF ASSINGMENT
The Consultant will be expected to work closely with CARE and ECIKS focal points in developing the MEAL plan for this project, to create a MEAL Plan that captures the most relevant and important project metrics and envisages its practical application throughout the implementation, by the project team. The consultant is expected to carry out and deliver on the following tasks:
- here. Please note that “recommended” gender points from the DCED Standard – i.e. points 1.5, 2.6, 3.5, 6.2 on pages 6 and 7 – are mandatory for this project.
DELIVERABLES
- Develop Theory of Change (ToC) in close collaboration with project team, outlining how different elements of the project come together to produce the project results and objectives?
- MEAL Matrix outlining impact/output/outcome indicators, data sources, data collection scheduling, MEAL tools and other necessary elements – all or most categories of information contained in the CARE MEAL guidelines (see Annex A)
- A clear and consensual definition of direct and indirect participants (“beneficiaries”) of the project.
- Inputs on data collection for the MSCA
- A completed baseline logframe, drawing data from the MSCA
- Training on general M&E, MEAL principles and the project MEAL system usage for entire project team.
- Short end of Assignment report in English, 2-3 pages.
Entire communication and all deliverables will be realised in English language. The project requires the highest level of written English in all its documents, which includes all Assignment outputs.
TIMEFRAME
The Assignment should be carried out within a limited number of working days from the date of the signing of the contract between CARE and the Consultant and 30 July, 2023. It is expected of the interested Consultant to propose, within their bid, the number of days envisaged as required for this Assignment, determined in a justifiable manner. The final number of consultancy days will be determined between CARE Project Manager and the Consultant within the consultancy planning communication and prior to the contract signing.
The Consultant will be paid for the amount agreed within the consultancy contract, through payments realised directly to the Consultant’s account, as follows:
Deliverables will be reviewed and certified as satisfactory by the EnInGeP Project staff and the Donor of the action. All deliverables must be submitted in electronic version in relevant Microsoft Office formats.
REQUIRED SKILLS AND QUALIFICATION
The Consultant fulfil the following criteria:
COORDINATION
The travel to the target country is not envisioned for this consultancy, the entire Assignment will be done remotely, with/through the usage of the information compiled online and from the project gathered data (mainly through MSCA).
All requests for clarification, logistic details should be arranged with the implementation unit of the EnInGeP project to Mr. Dragan Peric, dragan.peric@care.org, Project Manager.
Interested candidates are invited to apply and should submit:
The Consultant shall suggest a daily fee/rate per day in Euros on the basis of the depicted task. The Consultant should make a breakdown of costs in the budget part of his/her proposal, including different activities and types of costs and present the total costs based on daily fee.
The candidates should apply via e-mail to care.kosovo@care.org not later than 6th of April 2023 with the subject: Application for the position: Key Short-Term Expert for design of M&E system
References must be available on request. Only candidates with a suitable offer will receive a confirmation on receipt of their application. Only short-listed candidates will be contacted.
ANNEX A : CARE MEAL Approach, Principles and Operational Standards
| CARE INTERNATIONAL Monitoring, Evaluation and Learning Group |
CARE INTERNATIONAL
MONITORING, EVALUATION, ACCOUNTABILITY AND LEARNING
Approach, Principles and Operational Standards for Projects and Initiatives
(Update, July 8 2019)
THE FOUNDATIONS OF CARE’S MEAL APPROACH. 9
MEAL OPERATIONAL STANDARDS. 12
1. Design your MEAL system based on a clear theory of change and evidence needs 12
4.1 Monitoring outputs and participants 14
7. Use your MEAL system to continuously read the context and adapt to it 18
KEY QUESTIONS AS YOU DESIGN AND CONTINOUSLY ADJUST YOUR MEAL SYSTEM.. 18
| Foundations of CARE’s MEAL Approach Reality is complex and dynamic.Lasting change does not follow a linear timeline or a single pathway, where multiple stakeholders interact and influence each other, as well as our interventions.There are constant adjustments in social, economic, structural, environmental or other dimensions that we must be critically aware of and adapt to.Lasting change is the result of CARE’s contribution, as well as other factors, actors and elements of context. |
| The questions we aim at answering in all of CARE’s projects or initiatives WHO are the specific populations (women, girls, men and boys) ultimately experiencing lasting change, and who are the other actors facilitating that change?WHAT changes are those populations experiencing?HOW and WHY are those changes happening and what role does CARE and other actors play in facilitating those changes? |
| MEAL principles ❶ are conducive to Accountability. ❷ are conducive to Learning and potentially to Multiplying Impact. ❸ are conducive to Adaptation. MEAL systems and practices ❹ balance purpose, methodological rigor and capacity. ❺ consider ethical implications. ❻ are dynamic and lead to action and are conducive to gender equality. ❼ contribute to CARE’s global evidencing efforts. |
| MEAL standards ❶ Design your MEAL system based on a clear theory of change and evidence needs. ❷ Have a clear definition of participants: direct/indirect participants and target/impact groups. ❸ Define a meaningful and manageable set of quantitative and qualitative indicators and/or questions for impact, outcomes and outputs in each participant group, and the methods to track them. ❹ Define the monitoring and evaluation moments and methods that best ensure robust and comparable tracking of outputs, outcomes and impact. ❺ Ensure your evidence can be translated into learning and support on the identification of potential for scale. ❻ Make your evidence accessible and ensure your MEAL practices are participative and responsive to feedback. ❼ Use your MEAL system to continuously read the context and adapt to it. |
CARE International is a global leader within a worldwide movement dedicated to ending poverty. CARE seeks a world of hope, tolerance and social justice, where poverty has been overcome and all people live with dignity and security. CARE works around the globe to save lives, defeat poverty and achieve social justice, putting women and girls in the center, because we know that we cannot overcome poverty until all people have equal rights and opportunities.
Tackling the underlying causes of poverty and social justice is at the core of CARE’s programmatic approach, and we have an organizational commitment to demonstrate how we contribute to bringing lasting change to the lives of poor and vulnerable people, both in our humanitarian and development work.
This document outlines the approach, principles and operational standards that guide CARE’s Monitoring, Evaluation, Accountability and Learning practices for projects and initiatives implemented around the world, which can also be applied to the work CARE does with and/or through partners.
The content of the document combines key elements from longstanding policy documents like the CI Program Principles, Program Standards, the CARE Evaluation Policy, and others; together with elements from most recent MEAL approaches and practices in development and humanitarian work. As a result, it provides with an updated view on how CARE defines and operationalizes MEAL.
The successful implementation of CARE’s MEAL approach, principles and standards will lead to:
- Incremental and sustained improvement on CARE’s Monitoring, Evaluation, Accountability and Learning practices.
- Improved organizational capacity to demonstrate contribution to lasting change, capitalize learning on successful practices but also on failure.
- Improved quality in the design and implementation of projects and initiatives, and their capacity to adapt to changes in context, or changes motivated by feedback from stakeholders.
- Greater ability to generate knowledge and influence our own work as well as the practice of donors, peer agencies and other stakeholders, through sharing our MEAL expertise.
- Improved data or evidence supporting CARE’s fundraising and marketing efforts.
THE FOUNDATIONS OF CARE’S MEAL APPROACH
Figure 1: Lasting Change as defined by CARE
The foundation of CARE’s approach to Monitoring, Evaluation, Accountability and Learning is the recognition that we work in very dynamic and complex contexts, where lasting social change does not follow a linear timeline or a single pathway, where multiple stakeholders interact and influence each other as well as our interventions, and where there are constant adjustments in social, economic, structural, environmental or other dimensions that we must be critically aware of and adapt to (figure 1).
Under these circumstances, our organizational ability to demonstrate the impact of our work and explain how we contribute to lasting change lays in the ability of CARE’s
| Social change understood as the overcoming of poverty, enjoying equitable opportunities for women and men, being part of inclusive development processes and being able to continuously transform in response to new hazards and opportunities. |
projects and initiatives to put dynamic MEAL systems and practices in place. That means, MEAL systems that continuously generate comprehensive explanations and evidence on the way we think about a situation or problem and its underlying causes; a process of desired social change; how CARE’s interventions contribute to that change and how other factors and critical preconditions take place in society in order for that change to happen. In summary, MEAL systems and practices become critical to “unpack” the WHO, WHAT, HOW and WHY of social change:
- WHO are the specific populations (women, girls, men and boys) ultimately experiencing lasting social change, and who are the other actors facilitating that change?
- WHAT types of changes are those populations experiencing?
- HOW and WHY are those changes happening and what role does CARE and other actors play in facilitating those changes?
Applying this approach also implies that CARE’s MEAL systems and practices put special emphasis on explaining social change and impact as a combination of our actions plus the influence of other critical factors that make a change process possible (contribution), and only when appropriate, our MEAL systems focus on purely explaining social change as fully attributed to CARE’s actions (attribution). Although explaining attribution can often be considered a more robust way to show evidence of impact influenced by a set of interventions, we strongly believe that CARE’s contribution to social change is influenced and enriched by multiple actors and contributing factors. Therefore, our potential to explain complex change and multiply impact is enriched when we are able to complement a rigorous causal analysis with the explanation of the many elements influencing change, and the role of different actors have in facilitating that change.
Important Note: CARE’s MEAL approach includes the following definitions:
- Long-term or ultimate outcomes – Impact: includes sustainable, significant and measurable changes in well-being, materialized in lasting changes on poverty and social injustice conditions of a particular population. Changes at the impact level are influenced by those factors directly addressed by a project or initiative, as well as other factors.
- Immediate and intermediate outcomes – Outcome: includes changes on individual behaviors (e.g. individuals putting into practice new knowledge, attitudes or commitments) and changes that are structural or systemic (e.g. policy changes, new practices in service provision), that can be seen in different populations. Outcomes are often a result of what participants do on their own, influenced by the actions of a project or initiative or other factors.
- Output: includes the direct results of activities implemented by a project or initiative. Outputs may refer to: a) The results of training, such as the number of women trained in improved nutritional practices, farmers in improved agricultural techniques, etc. b) Capacity building, such as the number of extension staff trained, water systems built, committees established, etc.; c) Service outputs, such as an increase in the number of program locations; d) Service utilization, such as the number of people fed, or number or patients treated. Outputs are the products a project or initiative generates through the implementation of its activities.
- Inputs: Includes the set of resources that are needed by a project or initiative in order to deliver its commitments. These include the human and financial resources, physical facilities, equipment, materials, logistics, in-kind contributions and operational polices that enable services to be delivered.
In CARE, we acknowledge the fact that different actors use different definitions, nevertheless, note that this doesn’t affect the way the MEAL approach, principles and standards are defined and can be applied. The table below provides an overview of terms that CARE uses and their equivalence to the terms used by others
| CARE | USAID | DFID/UN | EC | Global Affairs Canada | Foundations (Gates) |
| Impact | Goal | Impact | Overall Objective | Ultimate Outcome | Strategic Area |
| Outcome | Purpose | Outcome | Specific Objective | Intermediate / Immediate Outcomes | Project Goal |
| Output | Output | Output | Result | Output | Objective |
| Input | Activity | Input | Activity | Input | Activity |
The fundamental propositions behind our Monitoring, Evaluation, Accountability and Learning systems and practices in CARE include:
❶ MEAL systems and practices should be conducive to Accountability, bygenerating solid and accessible evidence that clearly and transparently explains CARE’s work, the reach of its actions (what we do, where we work and the people we reach) and CARE’s impact story (our contribution to impact and outcomes); and by deliberately setting up mechanisms to engage and involve multiple actors and incorporating the perspectives of women, men, girls and boys in decisions and actions throughout the life of a project or initiative. This includes accountability to participants, donors and many other stakeholders.
❷ MEAL systems and practices should be conducive to Learning and potentially to Multiplying Impact, by generating and documenting evidence that strengthens the organizational memory and expertise, plus energizes learning dialogues and the identification of successful models and/or opportunities for scale-up.
❸ MEAL systems and practices should be conducive to Adaptation, by tracking, interpreting and summarizing key data related to changes in social, economic, structural, environmental or other dimensions that a project or initiative should be critically aware of and constantly adapt to.
❹ MEAL systems and practices should always balance purpose, methodological rigor, technology options and capacity, by identifying the most appropriate combination of methods to address: purpose (contribution / attribution), evidence needs and uses, resources, capacity, technology requirements and other factors[1].
❺ MEAL systems and practices should always consider ethical implications and be conducive to gender equality, by ensuring honesty, consent and integrity of all MEAL practices and MEAL methods selected; always respecting the security and dignity of the stakeholders with whom CARE works; incorporating gender and power elements when monitoring and evaluating; generating evidence disaggregated by sex, age and other relevant diversity categories, etc.
❻ MEAL systems and practices should be dynamic and lead to action, by intentionally planning and executing various iterations of monitoring, evaluation, accountability and learning moments throughout the life of a project or initiative, and stablish a clear connection with informed decision-making spaces, and actions.
❼ MEAL systems and practices in projects and initiatives should contribute to CARE’s global evidencing efforts, by generating evidence and engaging on reflection around CARE’s collective reach and impact story, globally.
From a practical perspective, the definition of a solid MEAL system for projects or initiatives in CARE should include the application of the following MEAL standards:
Projects and initiatives are normally designed based on a holistic analysis of context and stakeholders, plus a theory of change or any similar type of comprehensive explanation of the desired changes, the different pathways to get to the desired change and causality. The core of your MEAL system should be designed to continuously test the Theory of Change of the project and initiative, being able to answer questions like the following:
- What are the key outputs and activities the MEAL system will track in order to inform if the implementation of activities is in the right track, and reaching the expected participants? (direct and indirect participants)
- What are the key qualitative and quantitative changes (impact and outcomes) the MEAL system will track in order to inform if CARE is contributing to significant and lasting changes? Which pathways of change and causality relationships will we track? Who are the actors we will focus on when tracking those pathways? (impact and target populations)
- What are the key risks and assumptions the MEAL system will track and review during implementation in order to ensure the project or initiative is responsive to the context? How will unintended consequences or emerging changes be part of the continuous testing of the theory of change?
- What elements of gender, governance and resilience will the MEAL system will track?
- Which learning questions will the MEAL system help answer?
Projects or initiatives normally have well defined impact group(s) – those individuals that will ultimately experience impact or lasting change and target group(s) – those individuals whose behaviors or actions will influence the realization of changes for the impact groups (see figure in next page). If your project or initiative has a theory of change or has done a stakeholder mapping, the identification of impact and target groups would normally come from there.
Important note here: When looking at impact and target groups in your project/initiative, please remember that they must be identifiable as individuals that can be described and counted. They cannot be identified in general terms like households, families, groups, communities, organizations or other.
Here some examples of questions that can help you confirm if you are looking at concrete individuals:
| The (various) target groups: Composed of individuals whose actions or behaviors generate outcomes (e.g. changes in attitudes; changes in norms, policies, etc.) that influence the advancement towards lasting change or impacts for the impact group. These are groups with which the project or initiative works towards facilitating change and making a difference for the impact group. Examples: community/religious leaders, civil society representatives, decision makers, government officials, private sector actors, extended family members, community members in general, etc. |
| The impact group: Composed of a well-defined group of individuals for whom CARE and partners seek to facilitate lasting change or impact. Examples: women and girls; children under 5; poor small farmer households; children in school age; families affected by a disaster or crisis. |
Once the project or initiative has clarity on who the impact groups and target groups are, the next step would be for the project or initiative to define how it will count/track and report on these participants. In CARE’s language, this is where the concept of participants REACHED and participants IMPACTED comes into play:
| Participants REACHED | Participants IMPACTED |
| participants, however, this classification greatly depends on the modality of implementation that the project/initiative adopts, and other factors. For detailed guidance on how to define participants REACHED for different modalities of implementation, please refer to this guidance note | , experiencing lasting change in their lives (e.g. households graduating from extreme poverty; families becoming food secure; children under 5 no longer stunted; women generating income and accessing education; women participating in joint decision-making in the household, etc.) whose changed behaviors are also part of lasting change (e.g. local leaders/men and boys rejecting intimate partner violence). |
Note that, within the definition of participants REACHED, we are just looking at tracking/counting and reporting on individuals that are involved in the activities of a project or initiative. We are not counting/tracking/reporting if/how these individuals are experiencing important changes in their lives, like impact/outcomes (e.g. a participant that is receiving training would be counted as REACHED and, unless there is an evaluation process to determine if the knowledge acquired in the training has led to an outcome or impact, this individual would not yet be counted as participant IMPACTED).
The tracking of participants impacted, normally needs to be supported by an evaluation process (external or coming from a solid monitoring of outcomes) and requires measurement of outcome or impact indicators that the project or initiative has defined since its design (see next standard).
Make sure to incorporate at least one of the CARE’s Global Impact and Outcome Indicators, together with any other supplementary indicator that is relevant or required (e.g. by the donor). Recommendation: Try to avoid creating new indicators or indexes.
Quantitative indicators or questions would regularly help you demonstrate WHO are the specific populations experiencing change? (e.g. women of reproductive age; policy makers) And WHAT changes are they experiencing? (e.g. increased safe births; improvements in policy that guarantees women’s access to quality SRMH services). Qualitative indicators or questions would regularly help you demonstrate HOW and WHY are those changes happening? What role does CARE and other actors play in contributing to those changes? (e.g. is the increase in safe births explained by CARE’s strategy and work with health centers and communities? Or is it the product of changes in how decision makers recognize the importance of women accessing quality SRMH services ant take action on their own?).
Based on the indicators and questions you select for tracking impact, outcomes and outputs, define the most appropriate combination of methodological approaches to track them.
Cost-effectiveness: If the project or initiative has then intention to track cost-effectiveness, this is where you will need to design a tracked record of expenditures utilization linked with the identified outcome indicators, in order to be able to demonstrate the cost-effectiveness of certain strategies or outcome areas.
Define the moments, tools and resources used throughout the life of the project or initiative to track outputs from all those key activities being implemented (e.g. health staff from health services participating in training). While collecting and analyzing data at this level, the MEAL system won’t generate explanations related to impact or outcomes but will regularly ask if all the activities and outputs are the most appropriate and if they are really setting the bases towards the expected outcomes and impacts.
Important considerations when monitoring participants:
- Participants are always individuals. Even if our projects or initiatives work with households, communities or institutions, these are always composed of individuals, therefore, should ultimately monitored as individuals.
- One individual can be reached by one or more project or initiative in one particular context. The monitoring actions should be aware of duplications with other projects or initiatives, and establish the mechanism to report data without double counting.
- Participants’ data should normally be disaggregated by sex, age and potentially by disability or any key criteria related to the problem or vulnerability the project or initiative seeks to address. Estimations based on statistical references (e.g. census) are not always the most accurate measure. If the disaggregation is made using estimates, the source of the ratio must be explained.
- In projects or initiative implemented in the course of multiple years, the total participants in a particular year should be cumulative and single counted (existing and new participants). Even though it is important to know the incremental process, participant’s information is not normally aggregated year by year.
Define the moments, tools and resources used throughout the life of the project or initiative, to track key behavioral changes in some actors or strategic elements that set the causal linkage between outputs and outcomes and impact. Outcome monitoring helps generating indicative information (qualitative and quantitative) of what’s changing and what’s not / what’s working and what’s not, as the project or initiative advances towards the expected outcomes. For example, what happens after health staff participates in training? Do their behaviors change? How does changes in behavior favor women’s access to SRMH services? Outcome monitoring can be a continuous action (e.g. performing participant observation or doing informal interviews constantly), or a periodic action (e.g. applying an annual questionnaire or survey). In all cases, outcome monitoring may or may not have the same levels of representativeness of an evaluation, nevertheless, it does provide with important indications of progress and learning around the way the project is progressing towards contribution to change, the appropriateness of the strategies used and the validity of the assumptions in the theory of change.
Important considerations when operationalizing outcome monitoring actions:
- Is the volume of data, the frequency with which data is collected and the moments in time monitoring actions are undertaken, and the most useful for the project or initiative?
- Are the monitoring actions (collecting, reporting or analyzing data) considering the availability of time and predisposition of project staff or to project participants?
- Will all the data generated by monitoring actions be used and disseminated, and will inform decisions on the implementation or the theory of change of the project or initiative? Note: If that is not the case, you may be collecting more data than you actually need.
Value for Money: If your MEAL system needs to look at “Value for Money”, the monitoring component should incorporate elements that look at other elements like economy (quality of inputs), efficiency (delivery of outputs), effectiveness (extent to which outputs are converted into outcomes and impacts) and equity (extent to which most vulnerable groups are reached) of the actions implemented by the project or the initiative.
Define the moments, tools and resources used throughout the life of the project or initiative, to objectively assess its relevance and fulfilment of objectives, its efficiency, effectiveness, impact and sustainability, and/or its worth or significance (based on the OECD/DAC definitions). Evaluations in CARE projects and initiatives can be carried out for different purposes and take a variety of forms (see descriptions below). Nonetheless, all evaluations need to provide with substantiated evidence of the changes that took place as a result of a project or initiative’s actions, and a plausible explanation of how CARE’s actions contributed to the materialization of those changes.
- Formative evaluations: carried out during implementation of a project or initiative, intended to improve a project´s performance, informing necessary adjustments of project in relation to project design, planning, resources, approaches and methodologies, and capturing lessons and promising practices that inform decision-making (e.g. real time/mid-term evaluations of any project or initiative).
- Summative or End-line evaluation: often carried out at the end of a project, intended to assess the extent to which expected outcomes have materialized and assessing its significance or relevance (end-line evaluations).
- Impact evaluations: carried out either during or after the implementation of a project or initiative, intended to demonstrate impact in a cause-and-effect manner to an intervention. In impact evaluations, the focus shifts away from what CARE is doing, to observe and track the changes that take place in the lives of the impact groups, and how these changes come about. Impact evaluation normally entails a step further than any other type of evaluation and implies a deeper look to the participants and the changes they experience, plus collaborating with others in order to explain how these changes were facilitated by the project or initiative. As a result, it directs all is attention to test the theory of change behind the project or initiative and demonstrate how CARE contributes to that.
Important considerations when operationalizing evaluations:
- Evaluations should provide with complete and comparable assessments of the before-after or with-without situation.
- Evaluations should assess desired as well as unexpected outcomes.
- Evaluations can be conducted or supported by qualified professionals who establish and maintain credibility in the evaluation context. However, CARE staff should be highly involved in the whole evaluative process from the very beginning, not only to guarantee ownership of the process but also to open opportunity to strengthen MEAL capacities and to learn.
- Evaluation results need to be processed and reported in multiple ways and addressing different stakeholder needs and purposes. Evaluation results should be accessible for learning and for encouraging the project and participants to rediscover, reinterpret, or revise their understandings, plans and behaviors.
Monitoring and Evaluation actions normally generate a great amount of data and evidence, therefore, can naturally contribute to a structured body of information and knowledge inside and outside of. However, data and evidence can only be useful for learning and for multiplying impact, when data and evidence is adequately organized, processed, analyzed, discussed and shared.
Important considerations to link monitoring and evaluation with learning:
- What is it that we want to learn from the implementation of this project or initiative?
- Will the data or evidence to be captured by the MEAL system support learning in general or advance critical learning on a particular issue?
- Will it potentially generate evidence for multiplying impact?
- Is the monitoring and evaluation data sufficient and relevant enough for that learning or will we need additional research in a particular area?
Note that we can’t learn every single aspect of our work. Prioritization in a learning agenda is critical.
CARE’s commitment to accountability implies that projects and initiatives promote transparency in their actions, information and decisions, encourage participation from different stakeholders to shape their work, and deliberately open channels for feedback and take action based on feedback.
Important considerations when linking monitoring and evaluation with accountability:
- Ensure your MEAL actions balance the moments for data/evidence collection with moments for actors to provide feedback to CARE, and make sure to connect this feedback to the appropriate instances, so that feedback is always followed by action.
- Define how and in which moments will MEAL staff, program managers and other CARE and non-CARE actors will engage and collaborate in all the different steps of generating and using data, analyzing and responding to feedback, as well as making decisions for adaptive management.
- Ensure the MEAL system embeds a feedback and complaints mechanism that is comprehensive and in line with global standards (e.g. Core Humanitarian Standards).
- Make sure the targeting strategy of the project or initiative and the definition of participants promote equity and address the needs of the most vulnerable groups.
- Make sure key information generated by your MEAL system is accurately reported and available for different audiences (examples: PIIRS data, the Reach and Impact Map, etc.)
- Make sure your evaluations are well documented andpublicly available in CARE’s Electronic Evaluation Library http://www.careevaluations.org.
Adaptive approaches are increasingly and undeniable relevant to address complexity in the contexts in which we implement projects and initiatives. Our capacity to adapt covers many other areas of organizational culture, structures, processes and capacities that go beyond MEAL purely. However, MEAL systems can be highly instrumental for adaptation.
Important elements to consider when linking MEAL to adaptive management:
- Your MEAL practices need to be agile and have the capacity to collect data, generate evidence, identify changes and generate recommendations more frequently.
- The MEAL system should include regular review points when monitoring and feedback data is assessed against the theory of change, so that adaptation can occur accordingly.
- Your MEAL system should dedicate considerable effort to rapid learning and very agile feedback, in order to inform changes.
- Your MEAL system needs to be flexible, adjusting indicators, methods, tools and resources based on potential changes of the overall design of the project or the initiative.
- Your MEAL system should be clearly linked with decision making instances, in order to make sure that data and evidence signaling need for adjustments are taken into action.
KEY QUESTIONS AS YOU DESIGN AND CONTINOUSLY ADJUST YOUR MEAL SYSTEM
Finally, as you apply the principles and standards in your MEAL system and once you have defined the best possible MEAL process for your project or initiative, make sure that – in every MEAL step you take – you consider the following:
[1] In the current MEAL debate, we may find inclination towards assessing attribution, sometimes without sufficient resources or without considering the sensibility of applying certain methods in certain context. Being clear on the methodological appropriateness helps solving challenges around this. For example, if an intervention is seeking to generate evidence of impact, plus also validate a model or innovation, the selection of evaluation methods will be influenced by CARE’s global program priorities, the nature of the intervention, the rigor required, donors requirements, uses of the evidence, the capacities in place and resources available, etc. Normally, this results in the combination of quantitative and qualitative methods.
