Immunisation is one of the most effective public health interventions, significantly reducing morbidity and mortality from vaccine-preventable diseases. Globally, immunisation programs have played a pivotal role in controlling, eliminating, or eradicating diseases such as polio, measles, and smallpox.1,2 Although vaccines are highly effective, the success of immunisation programs and overall health outcomes depend on the quality of service delivery.3–7

Ethiopia has made substantial progress in expanding its immunisation coverage over the past decades, with the government prioritising the expansion of primary healthcare services.8–11 However, while national immunisation coverage rates have improved, the quality of immunisation services remains a critical area of concern.8,12 Quality in immunisation can be assessed through two main dimensions: technical quality, which refers to the competence of service providers in delivering vaccines safely and effectively, and perceived quality, which reflects mothers’ and caregivers’ satisfaction and trust in the service. Both aspects are essential for ensuring high immunisation coverage and effectiveness.13–15

In Ethiopia, challenges such as inadequate infrastructure, workforce shortages, competency/skill gaps and logistical barriers often impact the technical quality of immunisation services.8 Moreover, cultural beliefs, misinformation, and communication gaps between vaccinators and clients can influence the perceived quality of these services.16 However, the quality of the immunisation session and the client’s perception of the immunisation service were not explicitly documented in Ethiopia. Assessing the technical and perceived quality of the immunisation service can provide comprehensive insight into the strengths and weaknesses in current immunisation quality, offering actionable insights for improving service delivery.

This study aims to evaluate the quality of immunisation services in Ethiopia through the dual lenses of technical and perceived quality, encompassing pre-, intra-, and post-administration immunisation practices, as well as the satisfaction of mothers and caregivers. By understanding the current status of service quality, we can identify critical areas for improvement, enhance vaccine delivery, and ultimately contribute to achieving higher immunisation coverage and better health outcomes in the country.

METHODS

Survey design

This study was part of a national immunisation program evaluation survey aimed at assessing various components of the immunisation program, including session quality and mothers/caregivers’ satisfaction. The evaluation encompassed both the technical competency of vaccinators, assessed through direct observation during vaccination sessions, and caregiver satisfaction, gauged via exit interviews with mothers and caregivers.

Study setting and period

The study included all regions and city administrations of the country, except the Tigray region, due to the conflict. The Expanded Program on Immunisation (EPI) is provided at all regions and city administrations, at Health centres, health posts, and hospitals, including some private health facilities. The study was conducted from March to July 2023.

Study population

Vaccinators were the study population for assessing the quality of immunisation sessions through direct observation. In addition, mothers/caregivers who came for child vaccination services formed the study population for assessing their satisfaction.

Sample size and sampling

The sample size for the study was calculated based on the health facilities across regions and city administrations in the country. The determined sample sizes complied with the minimum guidelines set by the WHO for cluster surveys.17 A two-stage stratified sampling method was employed to choose the health facilities. In the first stage, enumeration areas (EAs) were randomly selected from stratified sampling frames that categorised urban and rural EAs, which were prepared for each region and city administration by the Ethiopian Statistical Services (ESS).

In each region, both urban and rural enumeration areas (EAs) were represented according to their population sizes, with 21.4% of EAs from urban settings and 78.6% from rural areas. Health facilities situated within the selected EAs and those nearby were included in the study. Overall, there was a proportional distribution of health facilities, comprising 291 health posts, 316 health centres, and 76 hospitals across various EAs, regions, and city administrations. Each health facility hosted between three and five vaccination sessions. Using a combination of expert judgments and Guidelines for Implementing Supportive Supervision18 Recommendation three to five session observations (2971) and exit interviews (2955) per facility were conducted to ensure valuable insights about the vaccinators’ competency and mothers/caregivers’ satisfaction, respectively.

Data collection tools and procedures

A vaccination session observation checklist was used to assess the competency of vaccinators in delivering immunisation services. The checklist included key activity indicators designed to evaluate vaccinator practices across three stages: pre, intra, and post-administration. A total of 21, 19, and 2 activity indicators were used to assess the pre-administration, intra-administration, and post-administration practices of vaccinators, respectively. Data collectors (observers) were recruited based on their previous experience with immunisation programs. Having experience in immunisation supervision and having completed the Immunisation Trainer of Training (ToT) training, these individuals were used to recruit the data collectors. A total of 115 data collectors participated in the data collection process. Although we used trained and experienced session observers, we didn’t conduct an inter-rater reliability assessment. Additionally, five Likert scale questions (10) were developed based on literature reviews7,19,20 to assess the satisfaction of mothers and caregivers with immunisation services. These questions addressed various aspects, including waiting time, cleanliness of the session, waiting area conditions, opening hours, availability of vaccinators, respect shown by vaccinators, and the information provided by vaccinators to mothers/caregivers. The data was collected using the Open Data Kit (ODK) Collect toolbox to reduce data entry errors and obtain real-time data.

Measure of immunisation service quality

The quality of immunisation services was assessed through both technical and perceived quality measures. Technical quality was evaluated by observing vaccination sessions, with a focus on the competency and adherence of vaccinators to key immunisation practices. Vaccinators’ performance was assessed using selected indicators that covered pre-, intra-, and post-administration activities during the vaccination sessions. Additionally, the perceived quality of the immunisation service was assessed through exit interviews with mothers/caregivers who attended the child immunisation service, which collectively evaluated their satisfaction with the vaccination service.

Data analysis

The vaccination session observation practice percentage was computed and interpreted separately for each indicator. We calculated the performance percentage of vaccinators based on activity indicators for pre, intra, and post-administration practices. Additionally, the level of satisfaction of mothers/caregivers was assessed by computing the score for each of the five Likert scale questions independently. We then created an index by categorising “very satisfied” and “satisfied” responses as “satisfied”, while the remaining responses (very dissatisfied, dissatisfied, and neutral) were classified as “dissatisfied”.

RESULTS

Session observation and mothers/caregivers’ exit interview characteristics

A total of 2,971 session observations were conducted, with the majority (75.6%) from rural settings. The majority of session observations were conducted at the health centres (63.6%), followed by health posts (19.2%). In addition, 24.1% of the sessions were observed from the Oromia region, followed by Amhara (23.8%) (Table 1).

Table 1.Vaccination session observation characteristics by region, setting and facility type
Characteristics Total number of sessions observed Percentage
Regions
Addis Ababa 99 3.3
Afar 150 5.0
Amhara 706 23.8
Ben/Gumuz 148 5.0
Dire Dawa 76 2.6
Gambela 36 1.2
Harari 106 3.6
Oromia 715 24.1
SNNP 476 16.0
Sidama 202 6.8
Somali 149 5.0
Southwest Ethiopian Peoples 108 3.6
Setting
Rural 2246 75.6
Urban 725 24.4
Facility Type
Public Hospitals 511 17.2
Health Centers 1890 63.6
Health posts 570 19.2
Total 2971 100

A total of 2,955 mothers/caregivers were interviewed, 24.9% were from Oromia and 21.8% were from Amhara regions. The majority (77.3%) were under 30 years of age, with 64.4% being homemakers. Regarding religion, 44.2% were Muslim, followed by Orthodox (34.2%) and Protestant (20.4%). Most respondents lived in rural areas (62.8%) and were married (98.1%). Additionally, 64.8% had families larger than four members (Table 2).

Table 2.Socio-demographic and economic characteristics of mothers/caregivers interviewed for satisfaction in Ethiopia, 2023
Characteristics Frequency Relative frequency
Regions
Addis Ababa 107 3.6
Afar 145 4.9
Amhara 645 21.8
Benishangul Gumuz 119 4
Dire Dawa 101 3.4
Gambela 51 1.7
Harari 100 3.4
Oromia 735 24.9
SNNP 359 12.2
Sidama 233 7.9
Somale 246 8.3
South West Ethiopia Peoples 114 3.9
Age
Less than 30 2285 77.3
Above 30 670 22.7
Occupation
farmer 481 16.3
Merchant 199 6.7
Housewife 1902 64.4
Employee 230 7.8
Private business 69 2.3
Students 36 1.2
other 38 1.3
Religion
Muslim 1307 44.2
Orthodox 1010 34.2
Protestant 602 20.4
Catholic 12 0.4
Others 24 0.8
Residence
Urban 1099 37.2
Rural 1856 62.8
Marital status
Married 2899 98.1
Never married 18 0.6
Divorced 27 0.9
Widowed 11 0.4
Family size
Less than or equal to 4 1040 35.2
Above 4 1915 64.8
Total 2955

Pre-administration vaccination safety practice

Only four per cent of vaccinators conduct a complete and proper infant assessment before vaccination. In addition, only 11% of the vaccinators used different education materials for immunisation-related communication. The practice of delivering key messages about immunisation was also practised by only 7% of the vaccinators. Almost half the vaccination sessions were delivered by a single vaccinator. Additionally, the proper reconstitution of BCG and measles vaccines was performed by 58% and 66% of the vaccinators, respectively (Table 3).

Table 3.Vaccination session safety during the preparation/pre-administration phase across regions, settings and facility types in Ethiopia, 2023.
Characteristics Session place cleanliness Availability of the table Availability of the chair Availability of two/more vaccinators Correct BCG diluent Correct Measles diluent Correct BCG reconstitution Correct Measles reconstitution Proper client greeting Ensure privacy Key messages Communication Proper infant assessment
About Antigen Health benefits of antigens Adverse effects Card handling Appointment All key messages provided Allow asking Properly explain questions Use educational materials Review the child's immunisation card. History of AEFI/side effects Medical condition Complete a proper infant assessment
National 73 87 73 48 58 65 58 66 66 52 41 36 36 52 86 7 42 95 11 85 27 67 4
Regions
Addis Ababa 100 100 100 84 80 59 76 59 26 51 60 24 35 60 82 6 49 100 0 93 51 71 2
Afar 68 81 97 22 89 89 86 89 81 63 55 32 48 51 56 4 56 95 34 84 11 79 1
Amhara 69 80 79 46 56 67 55 67 66 60 60 50 50 49 85 15 43 94 14 87 44 63 11
Ben/Gumuz 71 73 77 62 41 56 53 60 81 69 77 74 59 43 77 12 32 91 5 78 25 68 5
Dire Dawa 70 84 84 41 37 51 37 50 32 41 13 15 18 49 87 1 4 100 1 79 0 61 0
Gambela 78 89 89 100 69 81 69 81 83 72 67 67 67 76 62 22 50 94 11 92 32 88 6
Harari 75 70 75 57 55 65 57 58 84 29 17 3 20 40 77 0 34 83 5 96 4 71 0
Oromia 82 93 91 44 56 68 57 68 66 52 33 34 24 53 89 4 41 96 6 85 25 64 3
SNNP 64 85 86 48 47 55 47 57 75 49 23 26 32 58 88 2 45 92 11 82 8 52 0
Sidama 69 93 93 62 75 75 76 77 46 36 2 9 22 48 97 1 44 96 13 87 48 58 4
Somali 64 96 90 34 51 62 46 60 72 46 65 31 36 57 97 4 46 98 11 83 10 81 2
Southwest Ethiopian Peoples 80 96 89 56 71 62 71 62 68 61 36 38 31 55 89 5 45 96 13 84 17 90 3
Setting
Rural 70 84 84 45 54 65 55 65 71 52 40 37 36 53 88 7 44 96 12 84 28 66 5
Urban 83 94 96 60 68 68 67 68 53 55 43 33 36 51 82 6 38 91 7 88 22 69 2
Facility Type
Public Hospitals 77 91 92 54 61 63 63 67 64 62 39 32 34 54 89 7 48 95 12 85 24 68 5
Health Centers 74 88 87 44 57 64 56 63 68 53 40 36 35 50 85 12 42 95 11 86 27 65 3
Health posts 65 76 80 56 60 75 60 75 63 42 47 40 41 57 88 13 38 95 11 81 32 69 6

SNNP=Southern Nations, Nationalities and Peoples

Intra-administration vaccination safety practice

The observation also showed that 9% and 11% of the vaccinators missed the correct route and site during BCG administration, respectively. In addition, 8% and 9% of the vaccinators missed the correct route and site during Measles administration, respectively. Seven percent of the vaccinators also didn’t provide the correct dose of the Rota vaccine. The finding also showed that seven percent of the vaccinators didn’t administer Penta and PCV vaccines with the correct route and site (Table 4).

Table 4.Vaccination safety practices during administration in Ethiopia across regions, settings, and facility types, 2023.
Characteristics Position and calm BCG (603) Measles (645) Rota (1548) Pentavalent (2155) OPV (1892) PCV (2049)
Correct dose Correct route Correct site Used 0.05AD syringe Correct dose Correct route Correct site Used 0.5 AD syringe Correct dose Correct dose Correct route Correct site Correct dose Correct dose Correct route Correct site
National 84 98 91 89 92 100 92 91 98 93 100 93 92 99 100 93 93
Regions
Addis Ababa 83 94 94 94 74 100 100 100 100 100 100 100 100 100 100 100 100
Afar 83 85 85 71 82 94 85 74 94 91 96 91 89 99 96 92 93
Amhara 89 87 87 89 99 100 93 93 100 91 100 89 88 99 100 89 88
Ben/Gumuz 99 100 100 100 100 100 100 100 97 100 99 100 99 100 100 100 99
Dire Dawa 70 100 100 100 100 100 100 100 100 98 100 98 98 100 100 98 98
Gambela 97 100 100 100 100 100 100 100 100 100 100 100 100 100 96 100 100
Harari 94 100 100 62 100 100 95 100 100 100 100 100 99 100 100 100 97
Oromia 82 95 95 95 87 100 94 93 98 94 100 94 92 100 100 95 93
SNNP 72 84 84 88 93 100 87 87 91 91 100 90 90 100 100 90 90
Sidama 84 98 98 98 100 100 89 89 100 93 100 95 94 100 100 94 93
Somali 80 89 89 83 89 100 96 96 100 93 100 97 96 100 100 96 96
Southwest Ethiopia 92 100 100 100 88 100 90 90 100 100 99 100 95 98 100 100 95
Setting
Rural 85 98 88 88 94 99 90 90 97 93 100 93 91 99 100 93 92
Urban 81 98 98 91 88 100 97 95 99 95 100 96 95 100 100 96 95
Facility Type
Public Hospitals 83 98 96 94 89 100 89 87 100 93 100 92 92 99 100 92 92
Health Centers 84 98 91 88 91 100 92 91 97 93 99 93 91 100 100 93 92
Health posts 82 97 89 87 98 99 95 94 98 95 100 95 95 99 100 94 94

SNNP=Southern Nations, Nationalities and Peoples

Post-administration vaccination safety practice

Eighty-five percent and 80% of the vaccinators avoid needle recapping and properly use the safety box, respectively (Table 5).

Table 5.Post-administration vaccination safety practice by region, setting and facility type, and service delivery approach in Ethiopia, 2023.
Avoid needle recapping Proper safety box utilisation
National 85 80
Regions
Addis Ababa 100 100
Afar 39 69
Amhara 92 82
Benishangul Gumuz 92 69
Dire Dawa 95 79
Gambela 97 89
Harari 74 44
Oromia 81 85
SNNP 92 71
Sidama 76 90
Somali 83 85
Southwest Ethiopia 93 88
Setting
Rural 84 79
Urban 90 83
Facility Type
Public Hospitals 86 84
Health Centers 86 80

SNNP=South Nation, Nationalities and People

Maternal/caregiver satisfaction with the vaccination service

The study showed that only 55% of the mothers/caregivers reported being satisfied with the vaccination service. Satisfaction varied across regions, ranging from 23% (Afar) to 70% (Harari) (Figure 1).

Figure 1
Figure 1.Maternal/caregiver satisfaction with vaccination service across regions, settings, and facility type in Ethiopia, 2023

DISCUSSION

This study reveals sub-optimal immunisation session quality and client dissatisfaction with the immunisation service delivery. Only 4% of vaccinators conducted a complete infant assessment before vaccination, and just 11% used educational materials for effective communication. Moreover, only 7% of vaccinators delivered key immunisation messages, and nearly half of the sessions were handled by a single vaccinator. Alarmingly, 9% and 11% of vaccinators failed to follow the correct route and site for BCG administration, while 8% and 9% did the same for measles. Additionally, 7% administered incorrect doses of the Rota vaccine, and similar errors in route and site were found with Penta and PCV vaccines. Eighty-five per cent of vaccinators avoided needle recapping, and 80% used safety boxes correctly. The findings highlight gaps in the quality of immunisation sessions, with low adherence to assessment, communication, and vaccine administration protocols. The study also noted that 55% of mothers/caregivers reported satisfaction with the service.

In this study, the practice of complete infant assessment is low, indicating a significant gap in ensuring the readiness of infants for vaccination. Hence, a thorough assessment is crucial for identifying potential contraindications or health conditions that could affect the safety of vaccines.21 Almost one in ten vaccinators used educational materials for communication, reflecting a missed opportunity to enhance caregiver understanding, trust, and engagement in the vaccination process. Without adequate communication, caregivers may not fully grasp the importance of vaccines or know how to manage post-vaccination care.22 Moreover, the delivery of key immunisation messages practised by less than one in ten of vaccinators indicates a serious communication gap. Poor sharing of key immunisation messages with mothers/caregivers may contribute to vaccine hesitancy, as caregivers are not fully informed about the antigens and related information.23 Additionally, with almost half of the vaccination sessions handled by a single vaccinator, this resulted in a reduction in service quality due to time constraints, high workloads, and a lack of attention to both clinical and communication practices.

In this finding, almost one in ten vaccinators didn’t adhere to the correct route and/or site for BCG and Measles vaccine administration. Improper administration can compromise the effectiveness of the vaccines and increase the risk of adverse reactions. Additionally, about one in ten of the vaccinators commit errors in correctly administer Penta and PCV vaccines with the right route and/or site, which affects the vaccines’ protection against multiple diseases, and improper administration may leave children vulnerable to infections.12 Approximately one in ten vaccinators administered the incorrect dose of the Rota vaccine, potentially compromising its efficacy and reducing the ability to achieve the desired immune response in infants.24 These findings are aligned with different studies done in Ethiopia, which also showed gaps in immunisation service quality.12 The identified intra-administration gaps might result from several factors, including the absence of standardised immunisation quality guidelines, insufficient training, high patient volumes, and workloads. Addressing these issues is essential to ensure the safety and effectiveness of vaccination service delivery.

Almost four in five of the vaccinators avoided needle recapping and properly used safety boxes. However, the remaining 20% vaccinators failed to avoid needle recapping and didn’t use safety boxes correctly, which represents a significant safety concern. Needle recapping is a well-documented cause of accidental needle stick injuries, and improper disposal of used syringes can expose healthcare workers, waste handlers, and the community to infectious materials.8,25

Another key finding was the low satisfaction of mothers/caregivers with vaccination services, with only half of the mothers/caregiver’s expressing satisfaction. This indicates that nearly half of mothers/caregivers were dissatisfied with immunisation services. This can erode trust in the immunisation service, leading to vaccine hesitancy and higher dropout rates.26 Poor immunisation service experiences, coupled with ineffective communication and misinformation, may further discourage caregivers from completing the immunisation schedule. When key immunisation messages are not properly conveyed, caregivers may misunderstand vaccine benefits, increasing susceptibility to misinformation and negative perceptions, ultimately reducing vaccine uptake and coverage. This implies that it’s crucial to improve service quality, enhance communication strategies, and ensure the accurate dissemination of information to boost vaccine uptake and coverage. This necessitates a comprehensive review of service quality, with a focus on reducing waiting times, enhancing facility cleanliness, and improving the overall experience for mothers and caregivers.7,19,27 Addressing these issues could not only improve satisfaction rates but also contribute to higher vaccination coverage and public confidence in immunisation programs.

CONCLUSIONS

The study highlights considerable challenges to the quality of immunisation services in Ethiopia. Only a few vaccinators undergo a complete and thorough infant assessment before administering vaccines, and a small portion of vaccinators utilise educational materials to support communication with caregivers. The practice of effectively delivering essential immunisation messages was also notably low among vaccinators. Moreover, there were issues with proper vaccine administration, particularly in ensuring that vaccines such as BCG, measles, Rota, Penta, and PCV were administered using the correct route, site, and dosage. In terms of post-administration safety practices, a relatively high proportion of vaccinators adhered to avoiding needle recapping and using safety boxes correctly. However, the overall client satisfaction with vaccination services remained low. These findings suggest a need to establish an immunisation service quality standard, fostering a culture of accountability, and comprehensive improvements in the training and supervision of vaccinators, with a focus on enhancing both technical skills and communication abilities. Moreover, improving communication between vaccinators and caregivers, particularly through the use of educational materials and clear messaging, is crucial to boosting trust and satisfaction in immunisation services. Such actions will enhance communication effectiveness and overall service quality, ultimately improving the experiences and health outcomes of mothers and caregivers.

ACKNOWLEDGEMENTS

We extend our appreciation to the CBMP consortium universities for their academic and technical support. We are grateful to Amref Health Africa for funding the study and to WHO, UNICEF, JSI, and PATH for providing vital logistical assistance. We also acknowledge the Ministry of Health, as well as all regional, zonal, and woreda health offices, health facilities, and health posts, for their cooperation in facilitating field activities. We would like to extend our sincere gratitude to the participating communities, caregivers, and health workers for their time and invaluable contributions to this research.

DISCLAIMER

The views expressed in this article are solely those of the authors and do not necessarily represent the official positions of the Ministry of Health or partner organisations.

ETHICS STATEMENT

Ethical approval for this study was obtained from the Institutional Review Board of Hawassa University (IRB/288/15). Additional support letters were obtained from the Ministry of Health and the Regional Health Bureaus. Written informed consent was obtained from all participating mothers/caregivers and vaccinators. Data were collected using an anonymous electronic data collection system to ensure confidentiality and privacy.

DATA AVAILABILITY

The data supporting the findings of this study are available from the authors upon reasonable request. The dataset is not publicly available due to institutional data-sharing policies.

FUNDING

This study received financial and logistical support from Amref Health Africa, UNICEF, WHO, JSI, and PATH. The funders had no role in the design of the study, data collection, analysis, interpretation of the results, manuscript preparation, or the decision to publish.

AUTHORSHIP CONTRIBUTIONS

GA, BFE, YA, BD, LD, AT, AA, MA, KG, GT, SS, BT, and KA conceptualised and designed the study, conducted data analysis, prepared the first draft of the manuscript, supervised data collection, validated the findings, and critically revised and edited the manuscript. MM, NW, MN, YL, MZ, and MA contributed to study conceptualisation, conducted formal data analysis, supported data curation and resource coordination, reviewed and edited the manuscript, and supervised data collection. All authors reviewed and approved the final manuscript.

DISCLOSURE OF INTEREST

The authors completed the ICMJE Disclosure of Interest Form (available upon request from the corresponding author) and disclose no relevant interests.

ABBREVIATIONS

AEFI – Adverse Events Following Immunisation
BCG – Bacille Calmette–Guérin
CI – Confidence Interval
EAs – Enumeration Areas
EPI – Expanded Program on Immunisation
ESS – Ethiopian Statistical Services
IRB – Institutional Review Board
LMICs – Low- and Middle-Income Countries
MoH – Ministry of Health
ODK – Open Data Kit
OPV – Oral Polio Vaccine
PCV – Pneumococcal Conjugate Vaccine
SD – Standard Deviation
SNNP – Southern Nations, Nationalities and Peoples Region
ToT – Training of Trainers
WHO – World Health Organisation