Impact assessment of the Project For Active Inclusion & Rights of Roma Women – FAIR III in the Western Balkans

Impact assessment of the Project For Active Inclusion & Rights of Roma Women – FAIR…

Period for conducting the assessment: February 08 – March 15, 2022  Deadline for application: February 3, 2022 Project/Programme Title: For Active Inclusion & Rights of Roma Women in the Western Balkans [...]

Period for conducting the assessment: February 08 – March 15, 2022 

Deadline for application: February 3, 2022

Project/Programme Title: For Active Inclusion & Rights of Roma Women in the Western Balkans (FAIR III)

Countries: Bosnia and Herzegovina, Republic of Serbia and Montenegro

Project Number: 8285-00/2019

Project duration: February 1, 2019 – March 31, 2022

  1. Introduction/Background

Introduction:

CARE started working in the Balkans in the early 1990s to provide emergency and medical assistance to people affected by crisis, violence and displacement that engulfed the region due to the Yugoslav wars. CARE today works in Bosnia and Herzegovina, Republic of Croatia, Republic of Serbia, Kosovo, Montenegro, Macedonia and Albania. Since the 2000s, CARE’s main focus is on socio-economic inclusion of marginalized groups and gender equality. Since 2015, CARE has also been providing emergency assistance to refugees stranded in the Balkans.

Through its Gender Equality Program, CARE addresses issues of Roma women and girls as well as youth. It engages men and boys as allies in preventing gender- based violence and promoting healthy lifestyles and works on deconstructing social and cultural norms around harmful practices of masculinity. It supports Roma women leaders and activists in policy dialogues, accessing and defending their rights, and in the implementation of projects that address vulnerabilities of Roma and other marginalized women. All initiatives foster a joint regional response to issues that are common across all Balkan countries in order to replicate best practice models that lead to improvements in policy frameworks discriminating against Roma people, and particularly Roma women and girls. Thereby, networking and exchange between civil society and governmental actors, at local, national and regional levels, as well as advocating with policy makers for structural changes, presents CARE’s key approach in the Balkans.

Background:

The FAIR projects have been implemented by CARE International Balkans, with its HQ in Sarajevo (Bosnia and Herzegovina) and six Roma women led and/or Roma focused CSOs partner organizations that were selected via a competitive process at the project start:

BiH

  • ‘Better Future’ from Tuzla
  • Otaharin from Bijeljina

Republic of Serbia 

  • ‘Romani Cikna’ from Krusevac, 
  • ‘Osvit’ from Nis
  • Roma Women Center Veliki Crljeni, Lazarevac

Montenegro 

  • ‘Center for Roma Initiatives’ from Niksic

The first and second phase of the ADA funded three-year FAIR – For Active Inclusion & Rights of Roma Women in the Western Balkans projects were implemented in the three above mentioned target countries from 2012-2018, building on the lessons learned and expanding in both geographical coverage and partners’/participant’s outreach. 

FAIR III overall objective is to contribute to the promotion and protection of human rights and gender equality among Roma (RAE) women and girls in Bosnia and Herzegovina, Republic of  Serbia and Montenegro. The project purposefocuses on empowering Roma women and girls to be free and able to exercise their rights to live a healthy, dignified life free from violence, inequality and discrimination with support from their partners, families and communities

CARE and its partners planned to accomplish this by implementing a wide range of activities leading towards tangible change at the individual, local and national levels with the involvement of civil society organizations and government representatives and institutions reflected in the following four mutually reinforcing components or outputs:

Output 1: Enhanced capacities of Roma CSOs, youth and key community actors to practice and promote gender equitable, healthy and non-violent lifestyle with help of tested models and approaches.

Output 2: Improved access to and provision of services for Roma, Ashkali, Egyptian (RAE) women and girls (in particular on SRMH, GBV and Education) is ensured through strengthening of the Roma CSOs and the existing participatory accountability community mechanisms

Output 3: Three national-level Roma women networks are active and contribute to the effective functioning of the regional Roma Women Balkans Network and increase efforts towards Post 2020 EU Roma Integration Agenda.

Output 4Roma women and girls, CSOs and Networks are part of the regional and global social movement initiatives promoting and advocating for gender equality and (minority) women’s rights.

The direct target groups of the project are the six local partner Roma women organizationsnational Roma women network members (as organizations and individuals) that are involved in the implementation and the other Roma organizations making up the Balkan Roma Women Network. The project will directly target 26.150 people in total – aiming at 85% Roma and over 60% women and girls; at least 6,000 young people (Roma and non-Roma) and community members in the target locations will be part of the target group. 

Please find the Project Log frame attached in Annex 1 of these ToR.

  • Purpose 

This Impact Assessment will evaluate the overall accomplishment of the FAIR III project.

This comprehensive effort will serve to identify lessons learned and inform CARE Balkans future work, accountability and inform decision-making when it comes to CARE´s Gender Equality program’s focus on empowering Roma women through building agency, building relations and transforming structures.

This Impact Assessment End of Project Evaluation is expected to be conducted in the period February 08 – March 15, 2022.

  • Objective 

The overall objective of the Impact Assessment is to assess the results achieved based on OECD-DAC evaluation criteria (relevance, coherence, impact, effectiveness, efficiency and sustainability), with a particular focus on the impact and sustainability of the intervention.[1] . The Impact Assessment consultant will be expected to analyse the following points, as assessed necessary taking into consideration information available from previous evaluations/reviews: 

  1. the design and coherence (how well does the intervention fit) of the project/programme including the design of the log frame matrix/programme theory and present the underlying theory of change and its assumptions.
  2. the extent to which the project has already achieved its objectives and results or is likely to achieve them, including the extent to which the lives of the project/programme beneficiaries (women, men, girls, boys) has already been improved. Also, the extent to which supported institutions have already benefitted people. 
  3. the strengths and weaknesses in terms of planning, management, implementation and monitoring 
  4. the relevance and extent to which the issues of ADA’s social standards and environment as cross-cutting issues were applied.
  5. The results of this assessment will be used to inform the implementers, the donor and the involved stakeholders about the final findings and recommendations of the FAIR approach and to enable learning, improved planning and implementation of future CARE Balkans projects/programs in the region.
  • Subject and Focus

The topic of the Impact Assessment is to assess FAIR III’s performance based on the described criteria with the main focus being on the impact and sustainability: what changes have happened in the lives of the targeted people and organizations, as a result of the FAIR III efforts. 

The scope of the evaluation should consider the FAIR III internally conducted Baseline and Mid-term assessments. 

If the Covid-19 situation allows, the evaluation will take place in the three project target countries: Bosnia and Herzegovina, Serbia and Montenegro. The particular locations will definitely include the towns of key local Roma women partner organizations, but also additional locations where activities funded via sub-grant have been implemented. The final selection of locations will be agreed upon between the consultant team and CARE. The localities shall be selected based on the following criteria 

  1. evidence of successful examples of implemented actions
  2. diversity of approaches and/or localities (urban/rural, integrated/segregated with regards to ethnicity)
  3. potential problems on the implementation of the action/ local relations among diverse actors 

In terms of individuals, groups/organizations and institutions to be involved in this Assessment, representatives of all partner CSO, government institutions and key beneficiary groups that took direct part in FAIR III project implementation shall be included, either through surveys, individual interviews and/or focus group discussions.

  • Specific Evaluation Questions

The consultant is expected to provide a summary review of the end values as per the project log-frame at all the levels (output, outcome, impact) by using the data provided by the project. In addition, we expect the consultant to address the questions that would connect the key findings with the aspects of impact and sustainability, like for example:

  • What were the main changes brought about by the project identified by the participants: women, men, girls, boys (immediate impact) and at what level (agency, relations, structure)?
  • What stakeholders (like local Roma – women lead or women focused CSOs, local and national institutions, etc.) have benefitted from the project and how? What has changed for whom?

In the case of the benefits for local partners, organizational strengthening should be particularly considered, as a key aspect fostering impact and sustainability. Specifically, organizational and lobbying, communication and advocacy capacities; regional cooperation; networking with national stakeholders; visibility and general recognition of the organizations’ roles, locally and nationally. 

  • What were the major factors influencing the achievement or non-achievement of the project outcomes (due to the Covid-19 situation, this also covers the situations beyond the control of the project and how the challenges were mitigated)?
  • What activities and approach(es) seem to have been most impactful, in what terms and why as well as which of them have the highest sustainability potential and why is that so?
  • Approach and Methods 

The evaluation/review consists of several phases:

Contract and Kick-off meeting: Contract is signed, and a discussion of the assignment takes place. First documents, including available data, are provided to the evaluation team.

Desk Study: The evaluation/review team studies all necessary project/programme documents; re-construct and analyse the intervention logic/programme theory and theory of change and its assumptions. Existing data needs to be analysed and interpreted. 

Inception-Phase: In the inception report the consultant will describe the design of the evaluation and will elaborate on how data will be obtained and analysed. The use of a data collection planning worksheet or a similar tool is required. First interviews take place.

Data triangulation and quality control are key for this assessment and need to be discussed in the inception report.

The field trip will only take place upon official approval of the inception report by CARE.

Field-phase: Data needs to be gathered, analysed and interpreted. It is expected that the evaluation will include quantitative and qualitative data disaggregated by sex. This phase will include individual interviews and focus group discussions.

Presentation: Presentation of key findings (feedback online workshop) at the end of the field trip. 

Final Draft Report: Submission and presentation of final draft report, inclusion of comments from partners and contractor. 

Final Report: Submission of final report, see reporting requirements under point 9).

For the different phases it is expected that data and information will be obtained through different methods such as: analysis of documents, structured interviews, semi-structured interviews face-to face or by phone, group discussions, online-survey (if applicable), others.

All data collected needs to be disaggregated by sex. It is expected that the evaluation/review team will present concrete recommendations which are addressed to the specific stakeholders.

It is currently estimated that 15 people will need to be interviewed and at least additional 30 should be  included in focus group discussions (FGDs) in at least 7 locations in three countries.

  • Timetable

Optional: A total of 20 working days is currently estimated for this assignment.

ActionResponsibleDeadlineWorking days
 Submission of bid (electronically)Contractor 03.02.2022 
 Contract signed, and documents provided Contract signed between CARE and consultant 07.02.2022 
Kick-Off meeting  Meeting between contractor and consultant08.02.20221
Desk Study Consultant14.02.20223
First interviews Consultant18.02.20223
Development and submission of draft inception reportConsultant22.02.20222
Feedback to the inception reportCARE25.02.2022 
Submission of final inception report incorporating the feedback from CAREConsultant28.02.20221
Field Visit, interviews etc. and feedback workshopConsultant07.03.20224
Development and submission of final draft report Consultant11.03.20223
Presentation face to face or online of final draft report Consultant12.03.20221
Inclusion of feedback in final draft reportContractor14.03.20222
Submission of final evaluation report (hard copy and electronic copy) to CARE Consultant 15.03.2022 

8.The Consultant/Assessment Team

Key qualifications are: 

  • Relevant academic degree in international development, non-profit management, social studies or relevant fields.
  • A minimum of five years’ experience and expertise in conducting evaluations, assessments and research in the field/sector of social inclusion, women empowerment and gender equality.
  • Implementation of at least three evaluations in the last five years ideally in the relevant field and in the Balkans region.
  • Knowledge and working experience of/in the target countries with focus on Roma minority integration, in particular related to Roma women empowerment in the fields of education, health, anti-discrimination and prevention of gender-based violence.
  • Experience in project cycle management 
  • Experience preparing and analysing a theory of change
  • Experience and expertise in evaluating cross-cutting issues
  • Experience in social science methods
  • Excellent oral and written English skills (preferably also one of the Balkans languages spoken in the target countries)
  • Sound MS Office and IT skills

Cost

The Consultant shall suggest a daily fee/rate per day in EURO or BAM and the number of days predicted for the work per category. The consultant will propose a daily fee / rate per day in euros and the estimated number of days to work by category. The consultant should state all other costs through the daily fee (costs related to travel, accommodation, food, should be included in the budget). Payment will be in 2 instalments. First payment (30%) after feedback for the initial report. Final payment after acceptance of the final report by CARE.

The Consultant is required to make the daily work report in accordance with CARE Balkans policies (i.e., through daily time sheets).

The consultants must not have been involved in the design, implementation or monitoring of this project/programmed.

  • Reports 

The consultants will submit the following reports:

  • an inception report (up to 10 pages without annexes), 
  • and the final evaluation report (20-25 pages without annexes), the final executive summary and the results-assessment form (part of the reporting requirement) 

All reports need to be written in English. The executive summary needs to also be in one of the target country languages.

The executive summary should summarize key findings and recommendations (three to five pages) and needs to be submitted as part of the final draft report. 

The findings and recommendations of the draft final report and final report have to be structured according to the evaluation questions. An outline of the report’s structure needs to be agreed upon during the inception phase. 

The quality of the reports will be judged according to the following criteria:

  • Is the results-matrix format part of the report?
  • Does the report contain a comprehensive and clear executive summary?
  • Were the Terms of Reference fulfilled and is this reflected in the report?
  • Is the report structured according to the OECD/DAC criteria?
  • Are all evaluation questions answered?
  • Are the methods and processes of the evaluation sufficiently documented in the evaluation report?
  • Does the report describe and assess the intervention logic (e.g. logframe, program theory) and present/analyze a theory of change and its underlying assumptions?
  • Are cross-cutting issues analyzed in the report?
  • Are the conclusions and recommendations based on findings and are they clearly stated in the report?
  • Does the report clearly differentiate between conclusions, lessons learnt and recommendations?
  • Are the recommendations realistic and is it clearly expressed to whom the recommendations are addressed to?
  • Were the most significant stakeholders involved consulted?
  • Does the report present the information contained in a presentable and clearly arranged form?
  • Is the report free from spelling mistakes and unclear linguistic formulations?
  • Can the report be distributed in the delivered form?
  •  Co-Ordination/Responsibility

Ms. Jadranka Milicevic, FAIR III Project Manager will be the contact person for this evaluation/review.  E-mail: jmilicevic@care.ba

 Applications

Individual experts, companies or evaluation teams are eligible. The candidates (individual, company or teams) are expected to include in the application: 

  1. Cover letter 
  1. Technical Offer covering all aspects and tasks required in the ToR. The technical offer must include:
  2. Approach, methods and design including tentative work plan and division of responsibility (if team of consultants is applying) suggested by the applicant (3 pages)
  3. Professional profile of the evaluator(s) (CVs of all individuals included in the consultancy team)indicating contacts on reference persons from at least three former employees/contractors. 
  4. Former evaluation reports or other analytical outputs produced by the applicant.
  1. Financial Offer including cost breakdown. 
  2. The consultant(s) shall suggest the global price in EUR (per position if evaluation team applies) and the number of days predicted for the work. 
  3. Prices must in gross amount (free of all duties, taxes and other charges, including VAT). The amount of VAT may be shown separately.

Note:

If you are sending an offer as a physical person, state the net amount.

In case you are sending the offer as a legal entity, then state the amount with VAT, or if the legal entity is not in the VAT system, then only confirm it in writing.

Selection:

The applications will be assessed as follows:

  • Financial offer 30%
  • Technical offer 70%

Offers should be submitted by February03, 2022 by 5 pm CET to the following e-mail address: shalkic@care.ba and jmilicevic@care.ba. 

The selected will be notified in writing by February 06, 2022.

  1.  Annexes:
  2. Results-Assessment Form, to be filled in by the evaluation team
  3. Others can be included

Annex 1: Results-Assessment Form for Mid-Term and Final Project Evaluations/Reviews 

This form has to be filled in electronically by the evaluator/reviewer. No evaluation report will be accepted without this form. The form has to be included at the beginning of the evaluation/review report. 

Title of project/programme (please, spell out): 
Contract Period of project/programme:
ADC number of project/programme:
Name of project/programme partner: 
Country and Region of project/programme :
Budget of this project/programme: 
Name of evaluation company (spell out) and names of evaluators:
Date of completion of evaluation/review: 
 Please tick appropriate box:Evaluation/review managed by ADA/ADC Coordination Office x 
Evaluation managed by project partner: 
Please tick appropriate box:a) Mid-Term Evaluation           b) Final Evaluation           c) Mid-Term Review           d) FinalReview                                                                                                                                    x  
Others: please, specify:
Project Outcome  (Please, include as stated in the Logframe Matrix):  
For Final Evaluation/Review[2]: Project Outcome: To what extent has the project already achieved its outcome(s) according to the Logframe Matrix? Please, tick appropriate box Outcome(s) was/were:Fully achieved:Almost achieved:  Partially achieved:Not achieved: Please, also explain your assessment: What exactly was achieved and why? If not achieved, why not? (Please, consider description of outcome and relevant indicators)  
For Mid-Term Evaluation/Review[3]: Project Outcome: To what extent do you think the project will most likely achieve its outcome(s) according to the Logframe MatrixPlease, tick appropriate boxOutcome(s) will most likely be:Fully achieved:  Almost achieved:Partially achieved:Not achieved: Please, also explain your assessment: (Please, consider description of outcome and relevant indicators) 
Project Outputs: To what extent has the project already achieved its outputs[4]according to the Logframe Matrix ? Please, tick appropriate boxesOutput 1 (Please, include as stated in the Logframe Matrix):  Output was:Fully achieved: Almost achieved:Partially achieved:Not achieved: Please, explain your assessment: (Please, consider description of output and relevant indicators)   Output 2 (Please, include as stated in the Logframe Matrix):  Output 2 was:Fully achieved: Almost achieved:Partially achieved:Not achieved: Please, explain your assessment: (Please, consider description of output and relevant indicators)  Output 3 (Please, include as stated in the Logframe Matrix):   Output 3 was:Fully achieved: Almost achieved:Partially achieved:Not achieved: Please, explain your assessment: (Please, consider description of output and relevant indicators)  In case there are more than three Outputs please, state as above. 
Impact/Beneficiaries: How many women, men, girls, boys and people in total have already benefited from this project directly and indirectly? Please, explainWhat exactly has already changed in the lives of women, men, girls, boys and/or institutions from this project? Please, explain:Which positive and/or negative effects/impacts in terms of gender can be possibly be attributed to the project? Please, explain:If applicable, which institutions have benefitted from this project/programme and how?
Mainstreaming cross-cutting issues:Gender: To what extent was gender mainstreaming included in the project? To what extent were the recommendations – if any- from the ADA internal gender-assessment considered and implemented? Environment: To what extent was environmental mainstreaming included in the project? To what extent were the recommendations – if any- from the ADA internal environment-assessment considered and implemented? Which positive and/or negative effects/impacts in terms of environment can be possibly be attributed to the project? Please, explainSocial Standards: To what extent were the social standards monitored by relevant partners? Have any issues emerged? Please, explain   
Overall/Other Comments:   

[1] The 2020 ADA’s Evaluation Guidelines can be used as a good practice reference in the design and implementation of this assessment.

[2] Please, only fill in in case this is a final project evaluation/review.

[3] Please, only fill in in case this is a mid-term evaluation/review.

[4] In case there are more than three outputs, please, add them.

DOWNLOAD PDF