Methodology and reporting quality of 544 studies related to ageing: a continued discussion in setting priorities for ageing research in Africa

INTRODUCTION

The world’s ageing population is growing fast, and the population of older people is projected to reach 2.1 billion in the next three decades.¹ Similar trend of increasing elderly population has been observed in developed and developing countries, including the sub-Saharan African (SSA) region.¹ Although the population of older people aged 60 years in SSA constitutes less than 10% of the total population, a rapid demographic shift has been observed in this region - doubling the number of older people in some developed regions such as northern Europe by 2050.² By 2050, the population of older people in SSA is estimated to reach 163 million,² implying that more people will live longer than in previous generations. However, in this region, widespread poverty, lack of social security, and poor health outcomes characterize the living condition of the average older adult and hinder successful and healthy ageing.¹ Hence, strategies to address these issues are needed to prepare for the impending elderly population growth.

Despite the anticipated problems posed by the demographic shift, little consideration has been given to ageing research in SSA.³ Compared to the global north, the scarcity of data or research limits the formulation of viable policies to prepare the SSA countries for the future needs of the growing elderly population.⁴ However, efforts have encouraged research in the SSA region. Such efforts include, but are not limited to, stakeholders meetings organized by the National Institute in Ageing to advance ageing research in Africa in 2004⁴; establishing data collection centres for the Study of Global Ageing and Adult Health in three SSA countries – Ghana, South-Africa and Uganda (SAGE)⁵; and recently Emerging Researchers & Professionals in Ageing-African network’s articles on setting priorities for ageing research in the SSA.⁶ The latter group conducted a systematic mapping review of 512 ageing peer-reviewed studies, predominantly quantitative cross-sectional studies with a greater percentage of the included articles focused on HIV/AIDs, noncommunicable disease and cancer, followed by studies on the physical functioning of the older adults.⁶ Another important finding from this review was that studies, regardless of study design, increased tremendously over a decade, 316% increase for quantitative studies, 293% for qualitative, and 300% for mixed-method studies.⁶ While this proliferation of studies is a welcome development in preparation for the projected increase in the older adult population in the SSA region, little is known about the quality of these studies.

Quality assessment (also called quality appraisal or critical appraisal) systematically examines research evidence to assess its validity and relevance before using it for decision-making.⁷ As Sanderson et al.⁸ described, quality is an amorphous concept that typically entails two components: quality of reporting and methodology (i.e., what was done in a study’s design, conduct, and analysis) in research. Examples of methodological qualities include bias assessment, such as selection, performance, detection, attrition, and reporting selective biases; tools have been designed to assess these qualities.⁹ Reporting quality describes how well an article is written, often in a simple checklist.⁹ The EQUATOR (Enhancing the QUAlity and Transparency Of health Research) network described reporting guidelines are tools designed to ensure accurate, complete, and transparent reporting of research studies to support research productivity and usefulness.¹⁰ They have registered 256 guidelines on various health and social research study designs.^11,12 Methodology reporting is often challenging, especially when there is a poor quality of reporting. Therefore, authors are encouraged to assess articles with a tool containing sections for reporting and methodology quality, e.g., Downs & Black Checklist.¹³ In summary, the quality assessment provides information on the overall strength of evidence and methodological quality of a research design, highlighting the level of confidence the reader should place on the findings for decision making.

Appraisal of the methodological characteristics of ageing studies in SSA could help determine the strengths and weaknesses of these studies and how they could impact evidence and recommendations. High-quality empirical research is needed in SSA to know the current situation of older people, improve quality of life, promote healthy ageing, and inform policies. Therefore, this study aimed to assess the quality of studies on ageing in SSA.

METHODS

This paper is the second in a four-part series that aims to set priorities for ageing research in the SSA.⁶ The detailed methodology of the mapping review has been published elsewhere.⁶ Summarily, this paper followed Grant and Booth’s¹⁴ description of a mapping review to categorize existing literature to commission further reviews and/or primary research by identifying gaps in the literature. The quality of existing literature is essential to identify the literature gaps and possibly commission further studies.

Search strategy, study selection, and data extraction

Comprehensive search strategy, study selection and data extraction have been published elsewhere.⁶ Summarily, we searched seven databases, including PubMed, Excerpta Medica database (EMBASE), Cumulative Index to Nursing and Allied Health Literature (CINAHL), PEDRO (physical evidence database), Cochrane CENTRAL, PsycINFO and Web of Science, using medical subheadings (MeSH) in two concepts - Ageing and countries in Africa. The last search in the Kalu et al.⁶ was in February 2021, and we updated our search till December 2021. We included qualitative, quantitative or mixed-method studies focusing on older adults (55 years and older – 55years is considered to be older in the SSA region¹⁵), either directly (research conducted with or for older adults) or indirectly (e.g., studies that explored the experience of healthcare workers and students in providing care to older adults) (for full inclusion and exclusion criteria see Kalu et al.⁶). Multiple reviewers selected data (title/abstract/full-text screening) using predefined inclusion and exclusion criteria and extracted study data. Any disagreement was resolved in research meetings.

Current paper methodology – Quality assessment of the included studies.

This current study focused on the quality assessment of the articles in Kalu et al.'s⁶ review and articles retrieved from an updated search. Different tools were used for the quality assessment of the included studies. While the Down and Black Checklist was used to assess the quantitative studies,¹³ Case reports were assessed using Case Report (CARE) guidelines checklist.¹⁶ A 45-items tool described by Lundgren et al.¹⁷ was used to assess the qualitative studies, and mixed-method studies were assessed using the Mixed Methods Appraisal Tool (MMAT).¹⁸

Quantitative studies. The methodological and quality reporting of randomized controlled trials was rated using the Downs and Black Checklist (maximum score 28) and non-randomized trials using a modified checklist version (maximum score 24).¹³ The original checklist is comprised of 27 items over five domains [reporting (n=10 items), external validity (n=3), internal validity-bias (n=7), interval validity-cofounding (n=6), and power (n=1)]. Each item was rated as 0 (no or unable to determine) or 1 (yes), except for item 5, which was rated as 0 (no), 1 (partially), or 2 (yes).¹³ Articles received a score of 0 on the Power domain if the sample size was <300 based on population-based calculation for cross-sectional studies¹⁹ or if the authors did not state a prior sample size calculation for Randomized control trials (RCTs), to ensure that the study was adequately powered. The original version was used to rate the quality of RCTs and quasi-experimental studies, while the modified version²⁰ was used to rate cross-sectional, longitudinal studies and case-control studies. See Table 1 for modifications and scoring.²¹

Case report studies. The quality of reporting of case reports or case series was assessed using the CAse REport (CARE) guidelines checklist,¹⁶ consisting of 31 items, which was rated as 0 (no - if the item is not identified in the report) or 1 (yes - if the item is identified in the description); the highest possible score is 31. We rated the quality of reporting of the included studies as high (26-31), moderate (20-25), and >20 (poor).

Qualitative studies. We assessed the quality of the included studies using a 45-items tool previously described by Lundgren et al.¹⁷ We chose this tool because it provides additional items such as ethical issues, audit mechanism, relevance and transferability lacking in some other checklists used in assessing the quality of qualitative studies (e.g., Consolidated criteria for reporting qualitative research 32-item checklist). The Lundgren et al.'s¹⁷ 45 items composite grid checklist has five domains: research team and reflexivity (8-items), scope and purpose (2-items), study design (17-items), analysis and findings (14-items), and relevance and transferability (4-items). We rated all 45-items as 0 (no or not applicable) or 1 (yes); the highest possible score is 45. We rated the quality of the included studies as high (39-45 points), moderate (31-38) and low (≤30).¹⁷

Mixed-method studies. The Mixed Methods Appraisal Tool (MMAT), developed in 2006 and last revised in 2018,¹⁸ was used to appraise the methodology qualities of the mixed-method studies. The tool consists of 6 sections with related questions, of which assessors are required to answer – “Yes”, “No”, or “Can’t tell” with an option of writing a comment. Each of the six sections (qualitative, quantitative-RCT, quantitative-non-RCT, quantitative descriptive and mixed method) contains five questions, except the screening question has only two questions. For each study, we choose the appropriate quantitative study design and the qualitative study to appraise, in addition to the five questions mixed-method section. The five questions for mixed-method were grouped into Justification - Is there an adequate rationale for using a mixed-methods design to address the research question?; Integration - Are the different components of the study effectively integrated to answer the research question?; Interpretation - Are the outputs of integrating qualitative and quantitative components adequately interpreted?; Disagreements - Are divergences and inconsistencies between quantitative and qualitative results adequately addressed?; and, Adherence - Do the different components of the study adhere to the quality criteria of each tradition of the methods involved? For a sample of the MMAT, see http://mixedmethodsappraisaltoolpublic.pbworks.com/w/file/fetch/127916259/MMAT_2018_criteria-manual_2018-08-01_ENG.pdf. The overall numerical scoring of the MMAT is discouraged because it is less informative and fails to show the aspects of articles (qualitative vs quantitative) that are problematic.¹⁸ Instead, MMAT developers recommended that authors report the quality of each study design and the mixed method component and describe each study as a low or high MMAT studies. We classify a component (qualitative or quantitative) as low if they ≤ 60% of the MMAT criteria (i.e., if ≤ three questions were yes), and high if they > 60% (i.e., more than three questions were yes); this classification has been used previously.²²

The procedure of the quality assessment

We grouped reviewers based on their experience in quality assessment of quantitative, qualitative, or mixed-method studies. Author-reviewers were placed in each group based on their years of experience and the number of authorships in systematic review publications. That notwithstanding, we (Emerging Researchers & Professionals in Ageing - African Network - www.erpaan.org) conducted a two-day refresher training for all reviewers. Twenty-six author reviewers participated in the quality assessment of the included article. Because of the large volume of quantitative studies, 20 reviewers were assigned to assess the quality of RCTs and pre-post studies (n=2), longitudinal studies (n=4), case-control studies (n=2), cross-sectional studies (n=10), and case report (n=2). The remaining six reviewers assessed the quality of qualitative studies (n=4) and mixed-method studies (n=2). We performed pilot testing of the quality assessment of each study design, and the inter-rater reliability ranged from 0.78 to 0.87 across the groups, indicating a moderate to the high magnitude of agreement amongst raters.²³ Because we could not get a uniform agreement among raters in the pilot testing, the quality assessment for each study design was done in pairs. Summarily, two raters independently rated: 15 RCTs and pre-post studies, 56 longitudinal studies, 14 case-control studies, 60 cross-sectional studies, 4 case reports, 37 qualitative studies, and 15 mixed-method studies. The mean score of the raters was used as the final rating, provided the rating was in the same category. For instance, rater1 total score for an RCT based on the Down and Black Checklist¹³ is 18, and rater2 scored 20 for the same study. The mean scores of rater1 and rater2 are reported because both scorings are within the scoring classified as good [17 - 22] in the Down Black Checklist.¹³ A third rater assessed the same articles if two raters’ scores were not in the same category. Across all studies, a third rater was involved in only ten cross-sectional studies. The mean score of two out of the three raters in the same category was reported.

Data extraction and synthesis

We extracted the meta-data of the included articles: the countries in which each study was conducted, the study design, study settings, sampling method and recruitment strategies, participants’ characteristics (sample size, sex, age), data analysis method and the main findings, and these were analyzed using descriptive statistics, such as mean and standard deviation (for continuous variables), median, interquartile range (for categorical variables), frequencies, and percentages. This information has been presented elsewhere.⁶ The inter-rater reliability - the level of agreement between raters for quality assessment of the included articles,²³ was calculated using Kappa statistics. Values ranged from less than 60 (weak) 0.60 - 0.79 (moderate), 0.80 - 0.90 (strong) and Above 90 (almost perfect).²³ Each range was calculated using frequency count and percentages. We perform all data analysis in STATA (c).

RESULTS OF THE QUALITY ASSESSMENT

Additional 32 articles were added to the original 512 articles, totalling 544. Article type include: RCTs and pre-post (n=15), longitudinal (n=122), case-control (n=14) and cross-sectional (n=300); 4 case reports; 74 qualitative studies and 15 mixed-method studies. Studies were conducted 23 countries include South Africa (241, 44.3%), Nigeria (n=88, 16.2%), Ghana (n=53, 9.7%), Uganda (n=32, 5.9%), Tanzania (n=26, 4.8%), Kenya (n=24, 4.4%). Other studies were conducted: Cameroon, Central Africa Republic (9 studies); Senegal (6 studies); Malawi (5 studies); Burkina Faso (4 studies); Angola, Ethiopia, Zimbabwe (3 studies); Democratic Republic of Congo, Gambia (2 studies) and Benin, Eswatini, Namibia, Rwanda, Zambia (one each); and more than one African countries (25 studies). See Appendix A for a complete reference list of all the included articles.

Quality assessment of quantitative studies. Details of the quality assessment for RCT and quasi-experimental studies (n=15) can be found in Table 2. Only one was rated as excellent.²⁴ Four studies were rated as good,^25–28 eight as fair,^29–36 and the remaining two as poor quality.^37,38 Table 3 provides the quality assessment results of the 122 longitudinal studies. We rated 27 studies as excellent, 71 studies as good, 20 as fair,^39–58 and three as poor quality.^59–61 The quality assessment for the 14 case-control studies is described in Table 4. We rated four articles as excellent,^62–65 five as good,^66–70 and five as fair.^70–74 Out of the 300 cross-sectional studies, 48 were rated as excellent, 171 as good, 63 as fair, and 18 as poor quality^75–92 (see Table 5). Among the 49 studies rated as excellent, only three studies have insufficient power to detect a clinically meaningful effect.^93–95

Quality assessment of case reports. Table 6 shows the quality assessment of 4 case reports; all were rated as moderate quality reporting.

Quality assessment of qualitative studies. We presented the quality assessment result of the 74 qualitative studies in Table 7. We rated 19 as high-quality qualitative studies, 37 as moderate-quality, and 18 as low-quality qualitative studies. The domain with the least score was ‘Research team and reflexivity’, with one-third of the studies scoring above average.^96–117 However, only three studies scored below average in the ‘study design’^85,118,119 and ‘analysis and findings’^85,119,120 domains.

Quality assessment for mixed-method studies. We also assessed the quality of fifteen mixed-method studies (Table 8). Most of the studies (n=12, 80%) were rated high MMAT studies, and three were low MMAT studies.^121–123 Across the three low MMAT studies, one had poor scores in the quantitative domain,¹²² and the other two had poor qualitative domains.^121,123 In terms of the MMAT appraisal criteria, four studies did not satisfy the Justification criterion,^121–124 three studies did not satisfy the Integration criterion,^121–123 and six studies satisfied the Disagreement criterion.^{121–123,125,126} Five studies did not satisfy the Interpretation criterion^{121–123,126,127} and Adherence criterion.^{121–123,126–128}

DISCUSSION

This article is the second in a Four-Part series⁶ to describe the quality (methodological quality and reporting quality) of ageing-related peer-review articles in the SSA. The methodological and quality reporting of published studies relating to ageing in SSA showed variable quality, albeit primarily good quality, suggesting room for improvement, especially for RCTs. The quality assessment provides information on the overall strength of evidence and methodological quality of a research design, conduct and analysis, highlighting the level of confidence the reader should place on the findings for decision making. It is important to note that, even though the quality is generally good, the quality assessment reflects more on the quality of reporting rather than the methodological qualities (bias assessment) used in the included studies. Because of limited word counts and lack of use of reporting guidelines across the included article, it was challenging to assess the methodological qualities of the included articles. The first step to allow for practical methodological qualities assessment is encouraging ageing researchers in SSA to adopt reporting guidelines for any study design (See https://www.equator-network.org, for different reporting guidelines). After achieving this (ageing researchers using reporting guidelines to report their article), we can practically assess the methodological qualities (biases) that could reduce the reader’s confidence in the findings for decision-making.

Previous studies have argued that poor-quality research, either quality or methodology reporting in the SSA, is somewhat related to poor quality data on surveys or objective measures, such as incomplete reporting.¹²⁹ Generally, the unreliability of the data may be caused by sampling in the developing world, such as convenience sampling, which limits generalization, thereby impeding the use of such information in making a national or regional policy decision. Although the issue of sample size may not be specific to the ageing research in SSA, sample sizes were generally small in most studies included in our review. This highlighted the importance of longitudinal studies with an extensive data set and its accessibility for SSA researchers. Although longitudinal studies on ageing in the SSA region are increasing, pockets of community data sets in different forms are extensive, as shown in the articles included in this review. Therefore, global organizations, such as World Health Organisation-Ageing Africa or HelpAge-Africa, should create a data depository specific to ageing studies and develop standards/requirements to encourage researchers in different regions of Africa to deposit their study data. To guide against depositing poor quality or incomplete data, researchers depositing their collected data should clearly describe how it is collected. In addition, a committee should be commissioned to oversee and manage the data deposition; such a committee could be housed in already established centres for longitudinal studies, for instance, the WHO - SAGE collection centres in Ghana and South Africa. We recommend a standard data deposition reporting guideline for researchers, as it is promising to guide the researchers in providing a detailed description of their data collection methods. Besides, this checklist will guide the committee in reviewing and deciding which data should be deposited. To encourage researchers, any data deposit will be compensated with a fee. These data should be made available for researchers to answer different research questions and promote the global concept of data accessibility.

Could the implementation of National ageing policies in some African countries or the training and research in geriatric or gerontological have influenced the quality of ageing studies in this review? While we cannot measure this quantitatively, we could argue that it may indirectly impact the quality of studies. Currently, eleven countries, including South Africa, Ghana, Mozambique, Uganda, Tanzania, Zimbabwe, Kenya, Ethiopia, Malawi, and Nigeria, have implemented ageing policies.^130,131 These countries relatively have the highest number of high-quality articles across all study design; for instance, Nigeria: ^62,93,108; South Africa: ^{24,64,117,132,133}; Tanzania: ^116,134,135; Mozambique: ¹³⁶; Kenya: ^137,138; Ethiopia: ¹³⁹; Uganda: ^140–142; Ghana: ^143–145. Even though Rwanda and Cameroon’s ageing policies are at the drafting stage, studies from these countries were not of high quality and further highlighted our assertion that implementing ageing policies could help improve the quality of ageing studies in SSA. Nevertheless, there is a need to critically evaluate the impact of ageing policies on the quality of ageing studies. Furthermore, SSA countries with training and research in gerontology and geriatrics, and funding, such as South Africa,^125,146 have the expertise to produce high-quality studies. This highlights the importance of geriatric and gerontological training and its potential impact on Africa’s quality of ageing research. Since training and research in gerontology and geriatrics are hardly supported by the governments in SSA,¹³⁰ strategies to ensure adequate and continuous training in gerontology and geriatrics are recommended.

We observed that majority of the RCT/quasi-experimental studies (n=12/15, 80%) were conducted in South Africa, including one rated excellent and two rated good. The remaining two quasi-experimental studies were conducted in Nigeria and Kenya and were rated good. We also noticed that South African studies with ratings ranging between excellent and good were published in journals whose impact factors were higher (0.66 - 4.38) than studies from other countries with similar ratings published in 0.51 - 2.61 journals. This finding raises some questions to be considered by researchers and higher education institutions in SSA. First, could the dominance of South African studies within the RCT category be a function of the researchers’ training or knowledge in this design? Second, do more South African studies having better ratings (excellent and good) imply that the researchers better know the methods and feasibility of conducting RCTs or quasi-experimental research? Third, could the low representation of some SSA countries and the absence of others within this study category imply a lack of training of researchers or inadequate knowledge in this study design? Fourth, could it also be a function of resource-setting in terms of funding available for research? For instance, high-resource setting (in the case of South Africa) versus low resource-setting (in the case of other SSA countries). These questions are important areas to explore to understand better or highlight the discrepancies in the quality reporting of ageing studies in SSA.

While RCTs are considered the highest level of evidence for decision-making,¹⁴⁷ the cost of planning and managing RCTs is often high, limiting the number of high-quality RCTs in the SSA areas. Scholars have argued that non-RCTs can also provide substantial evidence, especially if they are of high-quality study design.¹⁴⁸ Both RCTs and non-RCTs are important in examining cause-effect relationships between an intervention and outcome; however, non-RCTs cannot eliminate the possibility of mediating factors in the outcomes. Where funding is an issue in developing and conducting RCTs, we encourage ageing researchers to plan quasi-experimental studies employing and describing the methodology using a reporting guideline to assess the methodological biases efficiently.

We observed moderate to low-quality qualitative ageing studies conducted in SSA. These low reporting qualities ranged from not reporting qualitative study type, sampling, and poor strategies reporting to ensure rigour. This information is needed to allow readers a logical process that will add credibility to the findings informing policies. Most studies did not provide a thick description of their methodology, making the study’s replicability in another setting challenging; this further doubts the study findings limiting its use in making clinical and policy decisions. Member checking, peer debriefing, data triangulation, and reflexive statement of the research team’s characteristic and how it influences data collection and analysis were lacking in most qualitative studies. Qualitative ageing research in SSA is increasing and has been under intense scrutiny for its methodology and reporting, hindering its use in clinical practice.¹⁴⁹ Therefore, qualitative ageing researchers use some of the qualitative reporting guidelines, such as consolidated criteria for reporting qualitative research (COREQ),¹⁵⁰ standards for reporting qualitative research (SRQR),¹⁵¹ or Lundgren et al.'s¹⁷ 45 checklist, is encouraged.

While the debates on quantitative research’s relative merits versus qualitative research are unending,¹⁵² researchers are encouraged to use both. Remarkably, 80% of mixed-method studies were high MMAT studies, and for the remaining 20%, the deficiency mainly was in qualitative studies, highlighting the need for training of qualitative ageing researchers in the SSA region. We should not celebrate this success story yet, since only 15 mixed-method studies were included and assessed in this review—a focused review on studies that utilized a mixed-method approach is warranted to either support to refute this success story.

While this is the first study to assess the quality of ageing research in SSA, it has some limitations. Even though our study strategy is robust, we may have missed some articles since some of the national Journals in SSA are not indexed in PubMED or related databases,¹⁵³ and access to African Journal Online [https://www.ajol.info/index.php/ajol], which houses most journals in SSA region is limited. This study did not include articles published in languages other than French and English; some articles may have been missed in several African languages like Afrikaans and Hausa. While we conducted the quality assessment in pairs and used the mean scores for reporting, there is still the possibility of quality assessment bias since quality assessment tools are inherently biased because of their subjective nature. We believed that using quality assessment tools for specific study designs would have yielded different ratings. For instance, using COCHRANE Risk of Bias tools¹⁵⁴ for RCT would have yielded different quality ratings.

We conclude that the methodological and quality reporting of published studies on ageing in SSA show variable quality, albeit primarily good quality, against excellent quality. Studies with a large sample size are recommended, and qualitative researchers should provide a section on research team members’ characteristics and reflexivity in their paper or as an appendix. Since this is the first study that describes the quality of published studies on ageing in SSA, a repeat quality assessment should be performed in the next decade. This paper is Part 2 of Kalu et al.'s⁶ review, and Part 3 of the paper will focus on a review of longitudinal studies of ageing in SSA, identifying areas that require longitudinal studies to explain the cumulative advantages and disadvantages across life course trajectory for the older adults population in the SSA region.

Acknowledgements

We want to acknowledge other members of the Emerging Researchers & Professionals in Ageing - African Network.

Funding

None.

Authors contributions

All authors contributed to the conceptualization, data screening, inclusion and extraction and manuscript drafting. All authors reviewed and approved the final manuscript for publication.

Competing interest

The authors completed the Unified Competing Interest form (available upon request from the corresponding author) and declare no conflicts of interest.