Dyon Hoekstra1,2,3*†
Margot Mütsch4†
Annegret Borchard4,5Christina Kien6,7†
Ursula Griebler6,7†Erik Von Elm5†Eva Rehfuess8,9†Ansgar Gerhardus2,10†
Stefan K. Lhachimi1,2,11†- 1Research Group for Evidence-Based Public Health, Leibniz-Institute for Prevention Research and Epidemiology (BIPS), Institute for Public Health and Nursing Research (IPP), University of Bremen, Bremen, Germany
- 2Health Sciences Bremen, University of Bremen, Bremen, Germany
- 3Department of Special Needs Education and Rehabilitation, University of Oldenburg, Oldenburg, Germany
- 4Department of Epidemiology, Institute of Epidemiology, Biostatistics and Prevention, Faculty of Medicine, University of Zurich, Zurich, Switzerland
- 5Cochrane Switzerland Center for Primary Care and Public Health, University Center of General Medicine and Public Health, Lausanne, Switzerland
- 6Department for Evidence-Based Medicine and Evaluation, University for Continuing Education Krems (Danube University Krems), Krems an der Donau, Austria
- 7Cochrane Austria University for Continuing Education Krems, Krems an der Donau, Austria
- 8Institute for Medical Information Processing, Biometry, and Epidemiology, Faculty of Medicine, Ludwig Maximilian University of Munich, Munich, Germany
- 9Pettenkofer School of Public Health, Munich, Germany
- 10Department for Health Services Research, Institute for Public Health and Nursing Research, University of Bremen, Bremen, Germany
- 11Department of Nursing Management, University of Applied Sciences Neubrandenburg, Neubrandenburg, Germany
Objectives: This study aimed to develop and apply a structured approach for prioritising topics for systematic reviews in public health, framed according to the readily applicable PICO format, which encourages the involvement of stakeholders’ preferences in a transparent matter.
Methods: We developed a multi-stage process, consisting of a scoping and two Delphi stages with web-based surveys and invited public health stakeholders in Switzerland to participate: First, respondents specified topics for different public health domains, which were reformulated in a PICO format by content analysis. Second, respondents rated the topics using five stakeholder-refined assessment criteria. Overall rankings were calculated to assess differences between stakeholder groups and rating criteria.
Results: In total, 215 respondents suggested 728 topics altogether. The response rate in the two Delphi stages was 91.6% and 77.6%, respectively. Most top-rated review topics focused on the effectiveness of interventions providing education to different target groups, followed by interventions to increase access to specific healthcare services.
Conclusion: Our approach encourages involvement of stakeholders in identifying priorities for systematic reviews and highlights disparities between stakeholders and between individual criteria.
Introduction
A major challenge in health research is that demand for funding usually surpasses the available resources. Health organisations carrying out or funding research must choose which research is to be prioritised [1]. Health research are often selected based on the priorities of researchers or a select group of experts, leaving out the interests of other relevant stakeholders, such as patients, policymakers and healthcare professionals [2–4]. Consequently, funded research can be of low priority to these potential users of research, making investments less efficient and increasing the risk of research waste [2, 5, 6]. Likewise, research needs of relevant stakeholders may not be addressed by funded research if they are not prioritised [7].
Several studies indicated that researchers sometimes overlook research needs of stakeholders in research priority setting (RPS) [4–6]. For example, patients or healthcare professionals prioritised non-drug treatments or management of social and emotional issues higher as researchers [8–10]. Recently, the interests of different stakeholders in health have been considered with varying degrees of involvement when determining what health research should be considered a priority [4]. This has led to a growing demand for structured processes for prioritising health research in which distinct stakeholder groups are involved [5, 11–13].
Furthermore, opinions, experiences and interests of individual stakeholders or the stakeholder groups they represent, lead to divergent views on which research areas should be considered worthy of priority [14]. This can lead to potential conflicts in decision-making [2, 9, 15]. Hence, a wide range of stakeholders and interest groups can utilise results from a structured RPS study as a valid basis for discussion [12, 13, 16]. Using a transparent process, can also make it easier for stakeholders to accept decisions that do not align with their own interest [17].
Involving a broad range of stakeholders in an RPS study that specifically focuses on public health is particularly challenging due to the multifaceted nature of the field, crossing different sectors and different levels, each containing a variety of sometimes hard to define and reach target groups. A few RPS studies have aimed to prioritise topics solely in the field of public health (e.g., [18–22]), with varying degrees of structuring and stakeholder involvement.
Additionally, the multifaceted character of public health poses a challenge when prioritising for evidence synthesis by conducting systematic reviews. Systematic reviews synthesize evidence from various studies, producing a type of evidence that stand out due to their methodological rigor, comprehensive coverage, structured synthesis, and significant role in evidence-based decision making, which makes them a highly valuable and reliable source of evidence in research, healthcare, policy-making, and other fields [23]. Systematic reviews are important for generating evidence-based answers to research questions, often about the effects of interventions or measures regarding a specific target group, by reviewing available scientific literature. Systematic reviews can offer a holistic view that can address the complexity that is often involved in public health issues and can therefore provide an evidence base for developing and implementing public health policies and programs [24].
Cochrane is an international network that prepares, maintains, and promotes the accessibility of systematic reviews. Although groups within Cochrane have used various methods for prioritising topics for systematic reviews in their research area, an inclusive, transparent and structured process to prioritise review topics is still lacking [19, 25, 26].
Furthermore, involving various stakeholders causes a challenge when prioritising systematic review topics because of the initial difficulty to develop specific systematic review questions, for example, based on the PICO framework. PICO stands for Population, Intervention, Comparator and Outcome [27] and is often used as a framework to formulate a systematic review question and to define inclusion and exclusion criteria for the review [27]. Since not all stakeholders know how to process or develop PICO-formatted questions or topics, an RPS for prioritising systematic review topics needs to enable this [28].
Therefore, we–on behalf of Cochrane Public Health Europe (CPHE), a subdivision of Cochrane Public Health–aimed to develop and apply a structured RPS approach in Switzerland that enables all stakeholders to prioritise systematic review topics for public health research.
We sought to investigate potential similarities and differences among the priorities of different stakeholder groups and aimed to examine which assessment criteria are considered most important for prioritising systematic review topics according to different stakeholder groups.
The study was approved by the ethical review board of the University of Bremen in Germany and the Canton of Zurich in Switzerland and we reported it here according to the REPRISE guideline [5].
Methods
For this study we used a multi-stage process, comprising a scoping stage and two Delphi stages:
In the scoping stage, we first established an advisory board of three subject-matter and methodological experts from Switzerland, Austria, and Germany. The advisory board members reviewed the study design and provided valuable insights for setting the scope of the study [29]. As such, three relevant domains of public health research based on the classification of the European SPHERE project [20] were identified: prevention, health promotion, and health services.
Based on these public health domains the following stakeholder groups have been deemed relevant:
• Research and/or higher education
• Administration and/or politics
• Health organisations representing certain target groups in the population
• Organisations representing healthcare professionals and institutions
• Health insurers
Conducting desk-based searches and using recommendations from the research team and the advisory board, we identified organisations in Switzerland within each stakeholder group. We then invited each organisation (149 in total) to nominate respondents from within their organisations to participate in the Delphi rounds. Of the 149 stakeholder organisations invited, 70 (47%) organisations nominated a total of 215 individuals (see Table 2).
Next, we applied a modified Delphi consisting of two online Delphi rounds for the prioritisation of the review topics. We used Lime Survey [30] for the online surveys (available in German, French and Italian).
Delphi Round 1
In the first Delphi round (March/April 2018), respondents were asked to propose up to five topics for systematic reviews which they thought to be relevant. To accommodate the varying familiarity of the stakeholders with systematic reviews, we asked them to specify their proposed review topics according to the different components of the PICO format in a stepwise manner by providing a template for each component. We did not ask the respondents to suggest a comparator (C), as this was deemed too challenging for respondents, most of whom had never worked with systematic reviews.
We asked the respondents to propose review topics across the three pre-specified domains of public health, i.e., prevention, health promotion, and health services. Two people (DH and MM) coded and aggregated the proposed topics using content analysis in the software MAXQDA [31] while a third person (SL) provided feedback. The aggregation through the content analysis ensured that the large number of redundant or overlapping suggestions were transferred in a practicable amount of sufficiently distinct review topics. We counted how many review topics were proposed in each public health domain to understand which target groups (population), interventions, and outcomes were considered most frequently.
Additionally, respondents had to select criteria that they considered most important for assessing topics for systematic reviews. For establishing an initial list of criteria, criteria taken from literature were discussed within the research team and with the advisory board. Based on the initial list, respondents could propose further assessment criteria, which we aggregated, and if applicable merged, with the initial list of criteria. We then examined which assessment criteria were selected most frequently. But we also ensured that the criteria used for rating, measured distinct dimensions to be able to sufficiently discriminate in the assessment of the review topics. Table 1 shows the five stakeholder-refined assessment criteria that were considered most useful for rating review topics in public health.
Table 1. List of stakeholder-refined assessment criteria after Delphi round 1 (Switzerland, 2024).
Finally, we asked respondents to rate their previous experience with and their attitude towards systematic reviews.
Delphi Round 2
All those nominated for participation in the first Delphi round were invited to participate in the second Delphi round (August/October 2020), independent of whether they had responded to the first round. Respondents were asked to rate the review topics based on the content analysis by applying the stakeholder-refined assessment criteria on a scale from 1 “totally agree” to 4 “totally disagree.” Each respondent had to assess approximately only one-third of the review topics, which were randomly assigned.
In this round, we additionally asked respondents to weigh the importance of the five stakeholder-refined assessment criteria on a 100-point scale.
We calculated the mean for each topic (named the “rating score”) for all five assessment criteria combined and for the five assessment criteria separately, to rank the topics accordingly [32]. Furthermore, the inclusion of a variety of stakeholder groups allowed us to compare the rating and ranking of the review topics and the criteria weighting by stakeholder group.
We used IBM SPSS Statistics (Version 23) software for data management and analysis.
Results
Characteristics of Participating Stakeholders
Of the 215 nominees, 197 (91.6%) started the online survey for the first Delphi round and 133 (67.5%) completed it (see Table 2). On average, the respondents took 25.5 min to complete the survey for the first Delphi round. Of the 197 respondents, 160 (81%) responded in German, 29 (15%) in French and 8 (4%) in Italian.
Table 2. Participation of organisations in nomination and of individuals in the Delphi rounds, by stakeholder group (Switzerland, 2024).
A majority of respondents in the first Delphi round described their work as being primarily “research and/or teaching” (n = 77, 39.1%). Followed by “administration and/or politics” (n = 37, 18.8%), “health organisations representing certain target groups in the population” (n = 36, 18.3%), “representatives of healthcare professionals and institutions” (n = 17, 8.6%); “health insurers” (n = 1, 0.5%); and “other” (n = 11, 5.6%).
In total 167 (77.7%) individuals participated in the second round, of which 128 (76.6%) completed the online survey. The respondents took 19.5 min on average to complete the survey for the second Delphi round. Only four respondents (2%) mentioned that they had not participated in the first round. Most respondents indicated that their field of activity was in “research and/or teaching” (n = 69, 41.3%), then “health organisations representing certain target groups in the population” (n = 39, 23.4%); “administration and/or politics” (n = 33, 19.8%); “representatives of healthcare professionals and institutions” (n = 15, 9.0%); health insurers (n = 3, 1.8%); and “other” (n = 7, 4.2%).
Respondents’ Experience With Systematic Reviews
Most of the respondents who answered the set of questions regarding their experience with systematic reviews (n = 138), stated some familiarity with systematic reviews and considered them essential for research: They knew what systematic reviews meant (n = 122, 88%) and had used them in their professional activities (n = 109, 79%). Two-thirds of the respondents relied on Cochrane for systematic review results (n = 91, 66%). A majority also agreed with the importance of systematic reviews in public health research (n = 84, 61%) and practice (n = 87, 63%); about half found the results of systematic reviews helpful for their own work (n = 69, 50%). Furthermore, approximately half of the respondents knew how to find relevant systematic reviews (n = 80, 58%) and less than half of the respondents (n = 59, 43%) indicated that they do not have difficulties understanding the essence of systematic reviews. Finally, less than half (n = 58, 42%) of the respondents had been involved in the process of producing a systematic review.
Results of the Content Analysis
The respondents suggested a total of 728 review topics in the first Delphi round that were compiled in a comprehensive list of 245 review topics, which we further aggregated into 27 categories across the three domains (see Table 3). For example, in the domain “prevention,” most suggested topics (n = 32) were assigned to the category “Interventions for screening and/or monitoring measures,” comprising often mentioned suggestions such as “More early screening to detect developmental abnormalities.”
Table 3. Frequency table of categories after content analysis (Switzerland, 2024).
We excluded 421 suggestions for review topics because they were: 1) defined as open questions or general statements (e.g., “Minimal case numbers: Quality feature or incentive for overtreatment?”), 2) too unspecific or not presented in PICO format (e.g., Development of new task allocations), or 3) defined only as outcomes (e.g., “Reduction of infant and maternal mortality”).
Although there was some overlap between the suggestions in the different domains of public health, the majority of suggestions for review topics could clearly be allocated to a single domain.
Rating of Review Topics
The overall rating results for all review topics and the 15 best-rated review topics per criterion, can be found in Supplementary Files 1, 2. The 15 best-rated review topics overall can be found in both Table 4, 5.
Table 4. Ranking differences between criteria for the top 15 review topics (Switzerland, 2024).
Table 5. Ranking differences between stakeholder groups for the top 15 review topics (Switzerland, 2024).
Among the top 15 review topics, most focus on interventions providing education to different target groups, followed by interventions to increase access to specific healthcare services. Mental health and substance use were the public health topics prioritised most often. Finally, the target groups considered most frequently are children and adolescents, specific target groups, and vulnerable groups.
We found large differences in the rating of review topics when examining the five assessment criteria separately. Of the 15 best rated review topics, not a single one was ranked in the top 15 in all five criteria individually. For example, the overall top priority “Improved access to prevention services for specific target groups” was ranked 3rd, 2nd, 5th, and 2nd, respectively, according to the criteria “Improving the health of the population,” “Health equity,” “Insufficient research to date,” and “Potential for innovative action”; however, it was ranked only 48th according to the criterion “Effect on public health if successful.”
Even larger differences in the ranking per criterion can be observed for many review topics from the top 15 list (see Table 4). For example, only four review topics that are in the top 15 according to the criterion “Insufficient research to date” are also represented in the overall top 15 review topics. Furthermore, the highest ranked review topics for the criteria “Effect on public health if successful” and “Potential for innovative action” are not at all represented in the overall top 15 review topics list and are even ranked among the worst 15 review topics according to two other criteria (i.e., “Insufficient research to date” and “Effect on public health if successful”).
Likewise, large differences in the rankings can be observed when comparing the overall rating between different stakeholder groups (see Table 5). Our results show that only the review topic with the best overall rating was rated in the top 15 by each stakeholder group. For all other review topics in the overall top 15 we found at least one stakeholder group that did not rate it in their top 15 list. However, the differing stakeholder group was not for every topic the same one, so none of the individual stakeholder groups was solely responsible for the observed differences.
The importance of using different assessment criteria and different stakeholder groups is further highlighted by the weighting of the criteria (see Table 6). Across all respondents, the criterion “improving the health of the population” received the highest weight of 29.49 (out of 100). The criterion “effect on public health if successful” was weighted lowest with 12.28.
Table 6. Weighting of assessment criteria by stakeholder group (Switzerland, 2024).
The high standard deviation of the overall weights per criterion (varying from 8.94 to 14.53) is mainly due to the large variability in the weightings of the different stakeholder groups. For example, (see also Table 6): Respondents from “research and/or education” (M = 29.65, SD = 1.89), “representatives of health professionals/institutes” (M = 36.43, SD = 3.82) and from “administration and/or politics” (M = 32.12, SD = 2.80) assigned the highest weight to the criterion “improving the health of the population”; whereas “Health insurers” weighted this criterion the lowest (M = 15.00, SD = 10.09). In addition, “health organisations representing certain target groups in the population” weighted the criterion “health equity” highest (M = 25.61, SD = 2.27), whereas “health insurers” rated the criterion “potential for innovative action” the highest (M = 30.00, SD = 7.95).
Discussion
We employed a multi-stage RPS approach, integrating a scoping stage and two Delphi stages with web-based surveys, to involve a broad range of stakeholders in Switzerland in the prioritisation process. The approach was developed on behalf of Cochrane Public Health Europe (CPHE), a subdivision of Cochrane Public Health, and was applied in Switzerland to prioritise systematic review topics for public health research. We intentionally limited the Delphi process to two rounds, contrary to many Delphi studies conducting at least three rounds [33]. Therefore, the results are an indication of preferences rather than a true consensus.
Several RPS studies and approaches focused on systematic reviews in health research in general or in certain fields within health research, with varying degrees of stakeholder involvement and use of assessment criteria [34, 35]. However, our structured RPS approach is developed to determine and prioritise systematic reviews solely for public health topics, which particularly 1) allows for the participation of a large panel of respondents from all relevant stakeholder groups in easy-to-conduct and easy-to-replicate online surveys, 2) implements assessment criteria specifically for public health and by public health stakeholders to allow for intra-criteria comparisons, and 3) assists all stakeholders in developing relevant systematic review topics in a PICO format before the prioritisation exercise and without needing prior expertise on systematic reviews.
In order to get a robust insight of what evidence needs exist in public health and not be limited only to the insights of a small group of experts or decision-makers, we aimed for a multi-perspective approach in our RPS study [4]. Therefore, after careful scoping to include all relevant stakeholder groups for this study, we applied a recruitment strategy that encouraged invited organisations to self-select respondents from within their own organisation to increase the likelihood that those invited will participate [36].
Due to our recruitment strategy and the accessibility of our RPS study as an anonymised online survey design we managed not only to recruit different stakeholders, we also achieved high participation rates in the first and second Delphi round of 91.6% and 77.7%, respectively (Cf. [37–39]). This represents a very high participation rate and demonstrates that with our approach it is feasible to involve many stakeholders in a structured RPS study using limited resources.
Applying our approach, we identified five stakeholder-refined assessment criteria that were deemed most important by the respondents for assessing topics for systematic reviews; albeit with differences in how these are weighted by different stakeholder groups. Previous RPS studies have listed assessment criteria that can be used in RPS and highlighted the importance of selecting multiple assessment criteria that fit to the specific context and that can sufficiently discriminate between the assessment of different review topics [11, 34, 40, 41]. The use of multiple assessment criteria also makes it easier for the respondents to rate the review topics as it provides clarity of what aspect of the review topic they are rating exactly. However, most RPS studies do not involve stakeholders in the selection of relevant assessment criteria [1]. We recommend to take the five stakeholder-refined assessment criteria, which we have applied (see Table 1), into consideration for other RPS studies in public health or related fields as they are based on the preferences of a broad range of public health experts.
Moreover, we found large disparities in the rating and ranking of the review topics when differentiating the results along the five assessment criteria and also along the different stakeholder groups.
The disparities between the stakeholder groups in the assessment of the review topics and in the weighting of the assessment criteria clearly show the conflicting interests of these groups and the need for including different stakeholder groups and the use of multiple assessment criteria in an RPS study [21]. The inter-criteria comparison and criteria weighting shows that distinct assessment criteria measure different dimensions of each review topic and can give insight on why a particular review topic is prioritised high or low [42]. The results corresponded to our assumption that the assessment of a particular review topic depends on the respective criterion applied (e.g., improving population health vs. insufficient research). Many RPS studies in health research do take multiple assessment criteria into rating and ranking options [11, 43], however inter-criteria differences are not presented.
Comparing the results of our RPS in Switzerland with other RPS studies that prioritised systematic review topics in public health [18, 19, 22, 43], it becomes clear that certain broader topic areas are shared top priorities, such as educational and health promotion interventions in certain settings, interventions related to access to healthcare services for vulnerable groups, and screening interventions. Furthermore, it can be observed that the top priorities are highly influenced by the geographical scope of the study. The RPS studies focusing on Nigeria [22] and West-Africa [43], prioritised several topics related to health burdens in their region, such as malaria, diarrhoea and maternal and child health, whereas our RPS and the studies of Doyle et al. [18] and Kingsland et al [19] who focused on global health issues, specifically highlighted interventions to improve mental and social health and interventions related to healthy diet and physical exercise.
Limitations
Although the main aim of our study was to develop and implement a structured approach for prioritising review topics in public health in general, the COVID-19 pandemic has had an impact on the results of our study: The first Delphi round was conducted in 2018 before the onset of the COVID-19 pandemic and the second Delphi round was conducted in 2020 in the midst of the pandemic. Therefore, it is likely that an updated run of the first Delphi round would yield some emerging topics that were not gathered during our first Delphi round. However, we do believe that many proposed and prioritised review topics, such as “Peer-to-peer education on health risks from drugs” and “Mental health resilience training,” are still considered highly relevant. Replicating this study would help to ascertain which topics are priorities for the short-term or for the long-term.
These long-term impacts and transformational potential of research can in general be a factor when it comes to the strength of RPS studies. It is often difficult to predict what impact particular research priorities are likely to have. Unilateral or short-term prioritisation could lead to long-term challenges or potentially transformative research being neglected. It is therefore important to carefully analyse the potential long-term impact of all the topics in the RPS [14] and to simultaneously be aware that prioritising specific topics can influence the long-term direction of research within a certain field. Our structured approach can be replicated easily and therefore has the potential to also investigate if the top priorities are considered important in the long-term.
Another factor that might influence the RPS study is that our approach has only been applied in Switzerland, with unclear applicability to other countries and world regions. Also, our RPS approach is focused on prioritising topics for systematic reviews along a PICO framework only. Further investigations are needed to find out if a similar structure is applicable for prioritising topics for other types of evidence or frameworks and if this would generate very different result.
A further limitation is that we used the PICO format which may be too narrow for those public health questions that aim at the evaluation of distal changes in multi-sectoral policies with indirect impacts on health [44, 45]. Hence, a potential extension would be to ask respondents additional and more detailed questions by expanding the PICO scheme by adding further dimensions, e.g., for setting or study design. However, this would increase the burden on the respondents participating in this study while also increasing the complexity of the analysis.
Finally, we have to stress that part of the differences found between stakeholder groups might be due to the differing, and in some cases small, group sizes. In general, the validity of our sample is constrained by the specific public health organisations we identified by purposeful sampling during our stakeholder search. This, for example, led to a large standard deviation of the criteria weighting for the stakeholder group “health insurers.” Our recruitment goal was not to have exact comparable group sizes, but to establish an extensive list of stakeholders from the field of public health. An alternative to this method would have been the use of an open call for participation in the Delphi stage. However, this comes with several disadvantages, such as uncertain participation rates, potential biases, and most of all unclear levels of respondent expertise. More research is needed to fully understand the distinct priorities by stakeholder groups and if patterns exist in how different stakeholder groups prioritise review topics.
Conclusion
Setting research priorities inherently involves value judgements and subjective decisions. These value judgements and subjective decisions may depend on the individual views, expertise and interests of the policymakers and other stakeholders [14]. The differences between stakeholder groups in the rating and ranking of the review topics and in the weighting of the assessment criteria that we found in our study clearly showed the differing interests of these groups. The contribution of a diverse range of stakeholder groups to a transparent RPS study is thus valuable in terms of content and should be encouraged through ongoing consultations with these groups in order to capitalise on the existing diversity [3].
Our applied modified Delphi technique allows for the inclusion of a large panel of respondents by presenting easy-to-conduct online surveys that reduces the workload for respondents to participate. The approach is designed to assist respondents in determining relevant systematic review topics in a PICO format without needing prior extensive knowledge on systematic reviews; thereby enabling the inclusion of all relevant stakeholders in the prioritisation process of topics for systematic reviews [46].
We believe that our RPS study can be replicated easily and likewise involve many relevant stakeholders in a prioritisation process. The online surveys and the rating results of this study are available in German, French and Italian, facilitating the adaptation and implementation of similar study designs in various settings.
Ethics Statement
This study, involving humans, was approved by the Ethical review board of the University of Bremen in Germany and the Canton of Zurich in Switzerland. The study was conducted in accordance with the local legislation and institutional requirements. The participants provided their written informed consent to participate in this study.
Author Contributions
All authors conceptualised the study. DH collected the data. DH, MM, and AB guided the data management and translation of the data. DH, MM, and SL analysed the data. DH was responsible for drafting this manuscript. All authors contributed to the article and approved the submitted version.
Funding
The authors declare that no financial support was received for the research, authorship, and/or publication of this article.
Conflict of Interest
The authors declare that they do not have any conflicts of interest.
Acknowledgments
We would like to show our gratitude to the advisory board members Milo Puhan and Brigitte Piso for their input during the course of this study and Thomas Heise for his support in the survey design. We are also very grateful for the time and effort that all the participants invested for our study: Not only the general response rate, but also the amount of proposed review topics and assessment criteria during the first Delphi round, exceeded our expectations.
Supplementary Material
The Supplementary Material for this article can be found online at: https://www.ssph-journal.org/articles/10.3389/ijph.2024.1606642/full#supplementary-material
References
1. Cartier, Y, Creatore, MI, Hoffman, SJ, and Potvin, L. Priority-Setting in Public Health Research Funding Organisations: An Exploratory Qualitative Study Among Five High-Profile Funders. Health Res Pol Syst (2018) 16(1):53. doi:10.1186/s12961-018-0335-8
2. McGregor, S, Henderson, KJ, and Kaldor, JM. How Are Health Research Priorities Set in Low and Middle Income Countries? A Systematic Review of Published Reports. PLoS One (2014) 9(9):e108787. doi:10.1371/journal.pone.0108787
3. Tomlinson, M, Chopra, M, Hoosain, N, and Rudan, I. A Review of Selected Research Priority Setting Processes at National Level in Low and Middle Income Countries: Towards Fair and Legitimate Priority Setting. Health Res Pol Syst (2011) 9(1):19. doi:10.1186/1478-4505-9-19
4. Grill, C. Involving Stakeholders in Research Priority Setting: A Scoping Review. Res Involvement Engagement (2021) 7(1):75. doi:10.1186/s40900-021-00318-6
5. Tong, A, Synnot, A, Crowe, S, Hill, S, Matus, A, Scholes-Robertson, N, et al. Reporting Guideline for Priority Setting of Health Research (REPRISE). BMC Med Res Methodol (2019) 19(1):243. doi:10.1186/s12874-019-0889-3
6. Chalmers, I, Bracken, MB, Djulbegovic, B, Garattini, S, Grant, J, Gülmezoglu, AM, et al. How to Increase Value and Reduce Waste When Research Priorities Are Set. The Lancet (2014) 383(9912):156–65. doi:10.1016/S0140-6736(13)62229-1
7. Reed, DA, Cook, DA, Beckman, TJ, Levine, RB, Kern, DE, and Wright, SM. Association Between Funding and Quality of Published Medical Education Research. Jama (2007) 298(9):1002–9. doi:10.1001/jama.298.9.1002
8. Crowe, S, Fenton, M, Hall, M, Cowan, K, and Chalmers, I. Patients’, Clinicians’ and the Research Communities’ Priorities for Treatment Research: There Is an Important Mismatch. Res Involvement Engagement (2015) 1(1):2. doi:10.1186/s40900-015-0003-x
9. Corner, J, Wright, D, Hopkinson, J, Gunaratnam, Y, McDonald, JW, and Foster, C. The Research Priorities of Patients Attending UK Cancer Treatment Centres: Findings From a Modified Nominal Group Study. Br J Cancer (2007) 96(6):875–81. doi:10.1038/sj.bjc.6603662
10. Tallon, D, Chard, J, and Dieppe, P. Relation Between Agendas of the Research Community and the Research Consumer. Lancet (London, England) (2000) 355(9220):2037–40. doi:10.1016/S0140-6736(00)02351-5
11. Bryant, J, Sanson-Fisher, R, Walsh, J, and Stewart, J. Health Research Priority Setting in Selected High Income Countries: A Narrative Review of Methods Used and Recommendations for Future Practice. Cost Effectiveness and Resource Allocation (2014) 12(1):23. doi:10.1186/1478-7547-12-23
12. Morton, KL, Atkin, AJ, Corder, K, Suhrcke, M, Turner, D, and van Sluijs, EMF. Engaging Stakeholders and Target Groups in Prioritising a Public Health Intervention: The Creating Active School Environments (CASE) Online Delphi Study. BMJ open (2017) 7(1):e013340. doi:10.1136/bmjopen-2016-013340
13. Viergever, RF, Olifson, S, Ghaffar, A, and Terry, R. A Checklist for Health Research Priority Setting: Nine Common Themes of Good Practice. Health Res Pol Syst (2010) 8(1):36. doi:10.1186/1478-4505-8-36
14. Lund, H, Tang, L, Poulsen, I, la Cour, K, Bjerrum, M, Nielsen, CV, et al. Lack of Systematicity in Research Prioritisation Processes—A Scoping Review of Evidence Syntheses. Syst Rev (2022) 11(1):277. doi:10.1186/s13643-022-02149-2
15. Prinz, F, Schlange, T, and Asadullah, K. Believe It or Not: How Much Can We Rely on Published Data on Potential Drug Targets? Nat Rev Drug Discov (2011) 10(9):712. doi:10.1038/nrd3439-c1
16. Barnieh, L, Jun, M, Laupacis, A, Manns, B, and Hemmelgarn, B. Determining Research Priorities Through Partnership With Patients: An Overview. Semin Dial (2015) 28(2):141–6. doi:10.1111/sdi.12325
17. Alonso-Coello, P, Schünemann, HJ, Moberg, J, Brignardello-Petersen, R, Akl, EA, Davoli, M, et al. GRADE Evidence to Decision (EtD) Frameworks: A Systematic and Transparent Approach to Making Well Informed Healthcare Choices. 1: Introduction. BMJ (2016) 353:i2016. doi:10.1136/bmj.i2016
18. Doyle, J, Waters, E, Yach, D, McQueen, D, De Francisco, A, Stewart, T, et al. Global Priority Setting for Cochrane Systematic Reviews of Health Promotion and Public Health Research. J Epidemiol Community Health (2005) 59(3):193–7. doi:10.1136/jech.2003.019547
19. Kingsland, M, Barnes, C, Doherty, E, McCrabb, S, Finch, M, Cumpston, M, et al. Identifying Topics for Future Cochrane Public Health Reviews. J Public Health (Oxford, England) (2021) 44:e578–81. doi:10.1093/pubmed/fdab287
20. McCarthy, M, Harvey, G, Conceicao, C, la Torre, G, and Gulis, G. Comparing Public-Health Research Priorities in Europe. Health Res Pol Syst (2009) 7(1):17. doi:10.1186/1478-4505-7-17
21. Hoekstra, D, Gerhardus, A, and Lhachimi, SK. Priority Setting to Support a Public Health Research Agenda: A Modified Delphi Study With Public Health Stakeholders in Germany. Health Res Pol Syst (2023) 21(1):86. doi:10.1186/s12961-023-01039-w
22. Meremikwu, M, Udoh, E, Nwagbara, B, Effa, E, Oringanje, C, Edet, B, et al. Priority Setting for Systematic Review of Health Care Interventions in Nigeria. Health Policy (2011) 99(3):244–9. doi:10.1016/j.healthpol.2010.11.004
23. Gough, D, Thomas, J, and Oliver, S. An Introduction to Systematic Reviews (2017). Sage Publications.
24. Petticrew, M, and Roberts, H. Systematic Reviews in the Social Sciences: A Practical Guide. Malden: Blackwell Publishing (2006). xv, 336-xv.
25. Bero, LA, and Binder, L. The Cochrane Collaboration Review Prioritization Projects Show That a Variety of Approaches Successfully Identify High-Priority Topics. J Clin Epidemiol (2013) 66(5):472–3. doi:10.1016/j.jclinepi.2012.03.015
26. Nasser, M, Welch, V, Tugwell, P, Ueffing, E, Doyle, J, and Waters, E. Ensuring Relevance for Cochrane Reviews: Evaluating Processes and Methods for Prioritizing Topics for Cochrane Reviews. J Clin Epidemiol (2013) 66(5):474–82. doi:10.1016/j.jclinepi.2012.01.001
27. Schiavenato, M, and Chu, F. PICO: What It Is and what It Is Not. Nurse Educ Pract (2021) 56:103194. doi:10.1016/j.nepr.2021.103194
28. Booth, A, Noyes, J, Flemming, K, Moore, G, Tunçalp, Ö, and Shakibazadeh, E. Formulating Questions to Explore Complex Interventions Within Qualitative Evidence Synthesis. BMJ Glob Health (2019) 4:e001107. doi:10.1136/bmjgh-2018-001107
29. Hoekstra, D, Mütsch, M, Kien, C, Gerhardus, A, and Lhachimi, SK. Identifying and Prioritising Systematic Review Topics With Public Health Stakeholders: A Protocol for a Modified Delphi Study in Switzerland to Inform Future Research Agendas. BMJ open (2017) 7(8):e015500. doi:10.1136/bmjopen-2016-015500
32. de Sá, JPM. Applied Statistics Using SPSS, STATISTICA, MATLAB and R. Berlin Heidelberg: Springer (2009).
33. Barrett, D, and Heale, R. What Are Delphi Studies? Evid Based Nurs (2020) 23(3):68–9. doi:10.1136/ebnurs-2020-103303
34. Fadlallah, R, El-Harakeh, A, Bou-Karroum, L, Lotfi, T, El-Jardali, F, Hishi, L, et al. A Common Framework of Steps and Criteria for Prioritizing Topics for Evidence Syntheses: A Systematic Review. J Clin Epidemiol (2020) 120:67–85. doi:10.1016/j.jclinepi.2019.12.009
35. Akl, EA, Fadlallah, R, Ghandour, L, Kdouh, O, Langlois, E, Lavis, JN, et al. The SPARK Tool to Prioritise Questions for Systematic Reviews in Health Policy and Systems Research: Development and Initial Validation. Health Res Pol Syst (2017) 15(1):77. doi:10.1186/s12961-017-0242-4
36. Schmitt, J, Petzold, T, Nellessen-Martens, G, and Pfaff, H. Priorisierung und Konsentierung von Begutachtungs-Förder-und Evaluationskriterien für Projekte aus dem Innovationsfonds: Eine Multiperspektivische Delphi-Studie. Gesundheitswesen (2015) 77(08/09):570–9. doi:10.1055/s-0035-1555898
37. Nast, I, Tal, A, Schmid, S, Schoeb, V, Rau, B, Barbero, M, et al. Physiotherapy Research Priorities in Switzerland: Views of the Various Stakeholders. Physiother Res Int (2015) 21:137–46. doi:10.1002/pri.1621
38. Li, T, Ervin, AM, Scherer, R, Jampel, H, and Dickersin, K. Setting Priorities for Comparative Effectiveness Research: A Case Study Using Primary Open-Angle Glaucoma. Ophthalmology (2010) 117(10):1937–45. doi:10.1016/j.ophtha.2010.07.004
39. Evans, C, Rogers, S, McGraw, C, Battle, G, and Furniss, L. Using Consensus Methods to Establish Multidisciplinary Perspectives on Research Priorities for Primary Care. Prim Health Care Res and Dev (2004) 5(01):52–9. doi:10.1191/1463423604pc186oa
40. Rudan, I, Gibson, JL, Ameratunga, S, El Arifeen, S, Bhutta, ZA, Black, M, et al. Setting Priorities in Global Child Health Research Investments: Guidelines for Implementation of CHNRI Method. Croat Med J (2008) 49(6):720–33. doi:10.3325/cmj.2008.49.720
41. Clavisi, O, Bragge, P, Tavender, E, Turner, T, and Gruen, RL. Effective Stakeholder Participation in Setting Research Priorities Using a Global Evidence Mapping Approach. J Clin Epidemiol (2013) 66(5):496–502. doi:10.1016/j.jclinepi.2012.04.002
42. Stebler, N, Schuepbach-Regula, G, Braam, P, and Falzon, LC. Use of a Modified Delphi Panel to Identify and Weight Criteria for Prioritization of Zoonotic Diseases in Switzerland. Prev Vet Med (2015) 121:165–9. doi:10.1016/j.prevetmed.2015.05.006
43. Effa, EE, Oduwole, O, Schoonees, A, Hohlfeld, A, Durao, S, Kredo, T, et al. Priority Setting for New Systematic Reviews: Processes and Lessons Learned in Three Regions in Africa. BMJ Glob Health (2019) 4(4):e001615. doi:10.1136/bmjgh-2019-001615
44. Rychetnik, L, Hawe, P, Waters, E, Barratt, A, and Frommer, M. A Glossary for Evidence Based Public Health. J Epidemiol Community Health (2004) 58(7):538–45. doi:10.1136/jech.2003.011585
45. Rehfuess, EA, and Akl, EA. Current Experience With Applying the GRADE Approach to Public Health Interventions: An Empirical Study. BMC Public Health (2013) 13(1):9. doi:10.1186/1471-2458-13-9
Keywords: public health, systematic review, priority setting, Delphi technique, stakeholder involvement
Citation: Hoekstra D, Mütsch M, Borchard A, Kien C, Griebler U, Von Elm E, Rehfuess E, Gerhardus A and Lhachimi SK (2024) A Structured Approach to Involve Stakeholders in Prioritising Topics for Systematic Reviews in Public Health. Int J Public Health 69:1606642. doi: 10.3389/ijph.2024.1606642
Received: 18 September 2023; Accepted: 06 August 2024;
Published: 21 August 2024.
Edited by:
Nino Kuenzli, Swiss Tropical and Public Health Institute (Swiss TPH), SwitzerlandReviewed by:
Two reviewers who chose to remain anonymousCopyright © 2024 Hoekstra, Mütsch, Borchard, Kien, Griebler, Von Elm, Rehfuess, Gerhardus and Lhachimi. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
*Correspondence: Dyon Hoekstra, aG9la3N0cmFAdW5pLWJyZW1lbi5kZQ==
†ORCID: Dyon Hoekstra, orcid.org/0000-0003-0677-2063, Margot Mütsch, orcid.org/0000-0003-0620-5376, Christina Kien, orcid.org/0000-0003-2158-962X, Ursula Griebler, orcid.org/0000-0002-9958-3714, Erik Von Elm, orcid.org/0000-0002-7412-0406, Eva Rehfuess, orcid.org/0000-0002-4318-8846, Ansgar Gerhardus, orcid.org/0000-0003-3637-7419, Stefan K. Lhachimi, orcid.org/0000-0001-8597-0935