Domains and Methods of Medical Device Technology Evaluation: A Systematic Review

Introduction

Medical devices (MD) are crucial healthcare technologies for the prevention, diagnosis, rehabilitation, and treatment of diseases, as well as in patient monitoring [1]. Recognizing the important role of these technologies in healthcare services, the World Health Assembly (WHA) adopted a resolution, WHA60.29 in May 2007, addressing the need to establish the management of these technologies [2]. In 2011, the World Health Organization (WHO) published a series of technical documents, including the thematic area of MD technology assessment, containing: i) assessment of MD needs, ii) MD procurement, iii) donations of MD, iv) MD inventory management, and v) MDmaintenance [3]. In 2014, the WHO member countries, through resolutionWHA67.23, recognized the enormous challenges in managing these technologies, particularly in low- and middle-income countries. The WHA67.23 addressed the need to establish tools to manage the use of healthcare technologies, as well as strengthen the link between technology assessment, regulation, and healthcare management [4].

In scenarios where economic resources are often scarce, the rigorous decision process for incorporating and disseminating these technologies represents a challenge for health systems. Health Technology Assessment (HTA) emerges as a process that seeks to verify whether a given technology is effective, safe and economically advantageous when compared to available alternatives. Since its inception, the field of HTA has been systematically developed by the U.S. Office of Technology Assessment, which published its first report on HTA in 1976 [5].

The HTA multidisciplinary process uses explicit methods to determine the value of a health technology at different points in its lifecycle. The HTA purpose is to inform decision-making aiming to promote an equitable, efficient, and high-quality health system [6].

The final outcome of HTA studies may vary depending on the country, the perspective adopted, the stakeholders involved and the decision-making context. The incorporation of new technologies without going through the necessary and critical HTA process can pose risks to the population and compromise the sustainability of the health system [7].

In general, HTA systematically examine various combinations of the following domains: technical performance, safety, clinical efficacy and effectiveness, cost, cost-effectiveness, organizational implications, social consequences, and legal and ethical implications of applying a health technology [8]. Choosing different domains for an assessment report depends on the specific context of the policy issue; one can focus on safety, clinical efficacy and effectiveness, and cost, such as in the pre-marketing phase, post-marketing phase, and even withdrawal of the product [7, 8].

In the specific case of medical devices, HTA has emerged as an important tool for supporting the core functions of effective and sustainable health systems and defining prioritization, incorporation and selection of MD [8]. However, unlike other health technologies, such as medicines, HTA processes in MD present gaps and weaknesses arising from the particularities and specificities of the MD sector. MD have challenged the HTA structures due to their constant technological evolution, strong dynamism of incremental innovation, learning curve, context dependence and wide heterogeneity.

There is a recognized lack of methods and tools to support HTA for MD, especially in the pre-commercialization phases. In the HTA process, there is also a lack of emphasis on organizational impact, equity, ethical issues, feasibility considerations, and acceptability by patients and healthcare providers [8].

A variety of aspects must be examined in the HTA process with a view to the implementation and adoption of these technologies in healthcare services. For highly complex MD, it is extremely important to consider the critical interdependence of these technologies. The best results cannot be achieved by investing only in a set of technologies. MD for diagnostic use must be combined with therapeutic capacity to have a significant impact on clinical management. There is a considerable risk that MD will not be used because the technology does not match the settings and resources available at the place of use [9].

The WHO technical series document “Health technology assessment of medical devices,” published in 2011 presents a set of essential elements for successful implementation of HTA projects at the national level, namely: good governance, adequate financing, adequate and good collaboration with partners. It emphasizes that health systems are strengthened when a HTA is integrated with human and material resources, in databases and sources, in decision-making, in the development of transparent policies, and linked to the global vision of equity and accountability [10].

In 2015, the WHO published the report “Developing an approach for using health technology assessments in medical product reimbursement systems,” which discussed the use of HTA in low- and middle-income countries (LMICs), including aspects of its use in relation to clinical practice, pricing policies, and reimbursement decisions. The document highlights that the use of HTA is adequately linked to other policy tools for efficient resource management, especially with regard to pricing and reimbursement policies [11].

Therefore, this review aims to characterize the domains and identify innovative methods for HTA studies with a focus on incorporating MD into healthcare systems.

Methods

Search Strategy and Selection Criteria

This is a systematic review following the Brazilian guideline for systematic reviews, and the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) hecklist. All parameters were pre-defined and registered in a protocol published on the OSF platform [12]. The project was registered and is available at [13].

The search was conducted in three stages. The first stage was an initial search limited to Embase and Web of Science online databases. This initial search aimed to analyze concepts included in index terms, author’s text keywords, title and abstract of the retrieved articles, as well as to help the research question elaboration.

The research question was organized using the PICO strategy (population, intervention, control, and outcomes): What are the main methods and domains that integrate HTA studies of MD from the perspective of technological incorporation in healthcare systems?

As a criterion for the research question, we used the definition adopted by the WHO for MD: “medical devices can be any instrument, apparatus, implement, machine, appliance, implant, reagent for in vitro use, software, material or other similar or related article, intended by the manufacturer to be used, alone or in combination for a medical purpose” [8]. We included studies in the review if they met the eligibility criteria described in Table 1.

Table 1. Eligibility criteria (Domains and methods of medical device technology evaluation: a systematic review. São Paulo, Brazil, 2024).

In addition to the criteria listed above, the studies were limited to publications in English, Spanish, or Portuguese languages. There were no restrictions on the study design type.

Based on the strategy PICO question above, the second search was conducted on 14 October 2022. Then, a targeted search strategy was performed at PubMed, Embase, Virtual Health Library (BVS), Cochrane Library, Web of Science databases, using controlled vocabulary and previously identified free terms through a targeted search strategy for each database. Additionally, gray literature studies were searched from the INAHTA and the WHO. On 3 May 2023, the search was updated. The search terms used included a comprehensive combination of terms related to medical devices and HTA, such as: (“Medical Technology” OR “Medical Device” OR “Equipment and Supplies”) AND (“Health Technology Assessment” OR “Biomedical Technology Assessments”). The search strategy by specific database, its execution date and the results, can be found in Supplementary Material S1.

Data Analysis

For the data selection processes, the online review management tool Rayyan was used. Two blinded authors (FT and DB) retrieved titles and abstracts (first-level) to establish the records suitability according to the inclusion criteria. Eligible articles were retained for the full-text article review. Reviewers’ discrepancies or disagreements were resolved by a third reviewer (LT).

For final review inclusion, once again the authors (FT and DB) independently reviewed the full-text articles. After completing the eligibility process, a consensus was established by the three authors.

The reference list of reports and articles identified as additional text studies sources was completed through the search of the INAHTA and WHO databases, using the free term “Medical Devices” and the same publication period, the last 5 years, in English, Portuguese, and Spanish languages.

The included articles and documents were then incorporated into the research management software Mendeley. Data from the studies included in the full-text were independently extracted by two authors using a standardized extraction form developed for the study in Microsoft Office Excel, according to the variables described below: Year, Title, Author, Journal, Country, Study Type, attribute, HTA Domains, HTA Method, Results, Knowledge Gaps, and Limitations. A third reviewer reconciled the collected data, and their categorization was done by consensus.

In the context of this systematic review of medical devices, the term “domain” was inspired by the EUnetHTA Core Model^®, which are: current use, technical, safety, clinical effectiveness, cost and economic evaluation, ethical analysis, organizational, patient & social, and legal aspects. The term “domain” here can be understood as the term to designate groupings of similar characteristics and specificities of MD. Domains can be composed of dimensions (subdomains) highlighted by their relevance in the context of MD HTA. Thus, during the complete reading of the selected articles, we sought to identify which of these domains were present, which others could be added, in addition to identifying the methods used in MD HTA analyses.

Dimension percentages were calculated considering the number of times they were referenced in relation to the total number of articles selected. Similar calculations were carried out for the methods table, considering in this case the method/tools/type of analysis identified in relation to the total number of articles selected.

The assessment of methodological quality and evidence confidence was performed by peers using specific tools according to the designs of the included studies. For articles where the authors themselves did not explicitly state the category of their study, we define the classification based on information from the methods used. Supplementary Material S2 presents the concept of each type of study that we used to define the typology of included articles. Systematic review studies were evaluated using the AMSTAR 2 tool. Narrative review studies using the SANRA tool. In cases where the methodology was similar to the experience report, or descriptive review or research survey were evaluated using the JBI Text and Opinion tool. The evaluation documents, containing the summary of the methodological approach and the classification of the type of study that we defined, are available in Supplementary Material S3 and were all considered to be of good quality.

Results

The searches in the databases resulted in a total of 5,771 records. After removing duplicates, titles and abstracts of 5,586 records were analyzed, of which 88 full papers were assessed according to the eligibility criteria. Additionally, 10 publications from WHO, 6 from INAHTA, and 3 from Google Scholar were identified, of which 2 publications met the eligibility criteria. A total of 41 studies were included in this review. Figure 1 presents the PRISMA flowchart of the search and study selection process. The list of 25 studies excluded during full-text reading, along with the reasons for exclusion, is available in Supplementary Material S4.

Figure 1. Flow chart depicting literature search and exclusion process (modified according to Preferred Reporting Items for Systematic Reviews and Meta-Analyses) (Domains and methods of medical device technology evaluation: a systematic review. São Paulo, Brazil, 2024).

The characteristics of the included studies are presented in Table 2. A significant number of the included studies were conducted in European countries (17/41), particularly related to the MedtecHTA Project (4/41), a specific European project on HTA for MD. Regarding the study type, the majority were reviews (35/41), including 18 descriptive reviews of HTA agency experiences, 14 narrative reviews, and three systematic reviews. There were also five survey research studies using stakeholders interview method and one WHO guideline.

Table 2. Characteristics of the included studies (Domains and methods of medical device technology evaluation: a systematic review. São Paulo, Brazil, 2024).

The results of this review were categorized into eight domains: 1) safety, 2) effectiveness, 3) health problem, current use of technology and innovation, 4) legal, 5) organization, 6) description and technical characteristics of technology, 7) costs and economic evaluation, 8) social participation (Table 3).

Table 3. Domains to be considered in Health Technology Assessment of Medical Devices (Domains and methods of medical device technology evaluation: a systematic review. São Paulo, Brazil, 2024).

Considering the domains of the HTA Core Model^® and our collected data, we add “innovation” within the domain of current use. We understand that ethical aspects would already be included in the evidence dimension when they are generated. Table 4 summarizes the current methods used in HTA studies in MD and their connection with some of the domains. The methods employed are not exclusive to any single domain; and can be applied across multiple domains.

Table 4. Methods associated with domains for evaluating medical devices (Domains and methods of medical device technology evaluation: a systematic review. São Paulo, Brazil, 2024).

In the articles analyzed, a wide variety of rearrangements of domains applied or desired in MD HTA were observed depending on the perspective adopted. In general, countries that carry out MD HTA employ similar domains, with differences residing in the healthcare systems’ care models.

The main domains and dimensions for MD HTA are well established, but the limited availability of clinical evidence (41% of the studies) is a significant difficulty. A recent approach to overcoming this gap is the use of Real-World Evidence (RWE) and post-marketing data in countries with robust techno-surveillance systems. This would assist in the transferability of technologies tested in clinical trials to real-life scenarios.

The “description and technical characteristics of technology” domain, represented by the characterization of the MD, received greater emphasis (73% of the studies), highlighting its importance as a starting point for evaluation. The learning curve associated was a significant factor in the score, suggesting a crucial consideration in evaluations for incorporation that is still little explored in terms of demonstrative data.

Legal aspects (68% of the studies) were identified as drivers for establishing quality regulations, but at the same time they were identified as weak in indicating transparent requirements for incorporation.

The organizational domain, which addresses the benefits for the health system, was highlighted in around half of the articles, highlighting, in this analysis, the importance of generating benefits not only for patients, but also for health systems. Although there is a preference for cost-effectiveness assessments (24% of the studies) in the Costs and economic evaluation domain, cost measures are still applied diffusely. However, the budgetary impact, often guided by the cost-effectiveness threshold, is often the decisive factor for incorporating a technology.

With regard to the life cycle dimension, disinvestment emerges as a prominent topic, although still incipient in discussions. Notable innovations include conditional reimbursement (10% of the studies) and consideration of the preferences of both patients and healthcare professionals (32% of the studies).

Discussion

In our research, we identified domains, dimensions and listed methods that support HTA studies, with a focus on incorporating MD into health systems. By definition, HTA seeks to inform decision-making based on the value of a technology at different points in its life cycle. This value is traditionally supported by the tripod: safety, efficacy and effectiveness. These three aspects were evident in the results of our research. The safety and efficacy domains were referenced with a frequency of 41%, and 44% of the included articles, respectively.

In the costs and economic evaluation domain, cost-effectiveness was the dimension most frequently reported among the included articles (24% of the studies). Legislation is the main driver of good HTA enforcement practices. It was highlighted in 68% of the studies, demonstrating its importance for filling gaps in studies involving MD.

The lack of international standardization of sufficient evidence for market access, less robust regulatory requirements, and the separation of HTA agencies from regulatory structures without alignment between their processes have weakened the conduct of HTA studies to obtain relevant clinical outcomes. Early HTA and initial dialogues involving HTA and regulatory actors to guide study design, sample size, comparator and outcomes selection are essential strategies. Balancing pre-market and post-market evidence, aligning with regulations, pricing policies, reimbursement, and procurement are important to reduce potential redundancies and increase synergy between process [14–17].

In the domain “health problem, current use of technology and innovation,” the evidence dimension (41% of the studies) obtained a similar score to the safety (44% of the studies) and efficacy (41% of the studies) domains. This demonstrates great interest in the topic of producing evidence for MD, considering the difficulties in conducting planned clinical trials, blinding, for example.

Independent initiatives seek alternatives to systematize this information. This is the case of IDEAL Collaboration, an international organization, which is developing an algorithm to guide the collection of pre- and post-clinical RCT data on MD involving complex surgical interventions [18]. The model adds important steps of preclinical and post-market evaluation, as well as indicating when RCTs should be complemented with other types of studies. Additionally, real-world evidence (RWE) studies that adhere to rigorous standards accepted by the scientific community can contribute to MD evidence [19, 20]. However, the transferability of such studies is more difficult to adopt [21, 22]. In light of this fact, the European Network for Health Technology Assessment (EUnetHTA) developed a transferability checklist: EUnetHTA Adaptation Toolkit [22].

Our study demonstrated that the “description and technical characteristics of technology” domain, dimension “characterization of MD,” is the starting point for evaluations, being reported in 73% of the found studies. Overall, studies discuss the need to establish specific methods for conducting HTA in MD. Due to the wide diversity of MD types, a single methodological guideline may not encompass all the specificities and intrinsic characteristics of the plurality of MD. Therefore, studies suggest using clustering criteria through technological characterization (risk class, mode of use, among others) as a strategy to make the process as standardized as possible.

The organizational domain (49% of the studies) was portrayed as as important as the aspect of the “health problem, current use of technology and innovation” domain. The implementation process of MD is also more challenging compared to other technologies, as they commonly require physical, human, and technological infrastructure requirements.

Therefore, the implementation strategy must be clear, robust, and adaptable for the adoption of new technologies in healthcare services. Communication tools are crucial to disseminate information in a diversified and intelligent manner to enable implementation.

Technical and assistance criteria for adoption and coverage should be explicitly stated [23, 24]. Provision for organizational support and stakeholder engagement is necessary. Additionally, establishing tools to monitor and evaluate the results and implementation of these technologies are important [25–27].

Regarding the “health problem, current use of technology and innovation” domain, which initially refers to clinical benefits, the identified gold standard method is the randomized controlled trial (RCT). Conducting this type of study involving MD is more challenging and difficult than with medications. Particularly for blinding and randomization, which may be ethically or technically infeasible.

Furthermore, studies discuss the difficulty of conducting clinical trials due to the characteristics of the MD production sector, predominantly composed of small and medium-sized companies with low expertise in conducting robust randomized clinical trials for MD, scarcity of funding for research, and less commercial reward for investment in R&D. There is a need for a joint effort involving academic, industry, and regulatory experts to assess the feasibility of designing and conducting RCTs [21, 28–32].

Regarding the domains of intrinsic characteristics of MD, iterative HTA approaches throughout the lifecycle of these technologies have been highlighted. The performance of a MD is often dependent on the context of use, infrastructure of the environment where it will be used, operator skill level, lifecycle and durability of components, supply chain and logistics, and interoperability between associated devices [23, 24]. Thus, many factors can influence the extent of benefits, and in the case of diagnostic and MD technologies that are co-dependent on other interventions, evaluating how subsequent patient outcomes are altered is important. These are also relevant points to consider in an economic evaluation.

In turn, the use of MD can impact the health organizational domain in two ways: interaction with the context can hinder its performance and benefits or modify the way services are delivered. To adopt these technologies efficiently, it is important to assess contextualization scenarios. The adoption of MD can be highly dependent on the organizational feasibility of healthcare services. The methodology, especially in economic evaluation studies, should consider intrinsic properties and characteristics specific to the type of MD under evaluation [33]. Iterative economic models can benefit decision-making according to the adopted context. The following tools and models for economic evaluations were mentioned in the reviews: CHEERS [14, 34], similar efficacy comparison, and cost calculation comparison method [35–37]. Basu and Egginton (2019) discuss the importance of documenting the intrinsic properties of MD on the CHEERS checklist to improve the transparency of economic evaluations [33].

There is a significant asymmetry of price information, as MD commonly do not have referenced prices through pricing tables. Dynamic pricing can occur due to factors such as new competitors entering the market, economies of scale and productivity gains, acquisition and contracting modalities, tax and tariff changes in the domestic and international markets, and fiscal benefits granted to institutional profiles. For technologies that require investment and capital expenses, the costs of irrecoverable investments should be weighed, such as the total cost of ownership associated with supply, installation, training, consumables, maintenance, and facilities are important considerations. The different financing models of MD can affect the estimates of economic evaluation studies and budget impact. Conditional coverage, risk-sharing agreements, coverage with evidence development (CED) have been discussed as strategies to minimize uncertainties in clinical and economic evaluation, both due to the learning curve effect, organizational impact, and technological life cycle of MD [18–20, 29, 32, 33, 35, 38–42]. Approaches for the use of multi-criteria decision analysis (MCDA) has also been reported [21, 43–45].

The need to establish criteria for prioritizing HTA demands in MD was addressed in the included studies [18]. The studies presented criteria for prioritization considering relevant aspects, such as: economic, social, organizational, ethical impacts, epidemiological relevance and uncertainty in clinical evidence [15, 46]. Criteria are discussed for various purposes, such as defining and developing risk sharing agreements [39], for divestment [47, 48], as well as criteria for establishing accelerated registries [32].

The main methodological approaches extracted from the included studies are described in a narrative form in the Supplementary Material S5.

Early assessment methods, such as horizon scanning (HS), are essential for an economic sustainability agenda to identify clinically beneficial and cost-effective new technologies, promote their early adoption, and ideally replace less effective or more expensive technologies and practices, as well as contribute to guiding the appropriate timing for conducting assessments [25, 28, 29, 49, 50].

The redundancy and duplication of HTA in MD studies can be exacerbate, especially by the characteristics of rapid development with constant modifications to the product, in addition to the wide possibility of indications for use. Hawlik et al. (2018) discussed the duplicity analysis of HTA MD studies in Europe, highlighting some redundancies, especially for HTA institutes in Spain, where identical technologies were evaluated by up to three different institutes. To avoid redundancy and avoid duplication of efforts, collaborative HTA studies are necessary through interest groups and similar activities that involve the sharing of scientific information [49].

To avoid redundancy and duplication of studies, shared databases and repositories with ongoing and completed studies are relevant. Collaborative activities can occur in the phases of literature search protocols, extraction tables, information on the description and technical characteristics of the technology, as well as in defining the health problem, executive summaries, and full reports. Involvement of stakeholders is necessary to ensure legitimacy, including regulators, payers and policymakers, research and academia, industry, patients and caregivers, healthcare service providers, and social organizations [25, 28, 49, 51, 52]. International collaboration is required to overcome inherent challenges in MD HTA [19].

Fragmentation between decision-making in the national HTA and processes of acquisition and implementation of technologies in local health services is practically inevitable due to the institutional structures. Due to the diversity of configurations available in the MD market, in addition to the specific requirements for local compatibility, installation, training, supply chain and maintenance, the centralized procurement process with a single payer is more challenging and complex when compared to medicines. Thus, after the decision to incorporate a new MD into the health system and for the proper provision of access, under equitable conditions, it is necessary to make efforts to coordinate the entire process to ensure the intended outcomes [15].

Strategies to reduce uncertainties resulting from the fragilities of clinical and economic evidence in decisions on incorporating MD into healthcare systems have been widely discussed. There are variations in definitions, CED, risk-sharing agreements and conditional coverage. Robust databases capable of generating data on the use of these technologies are essential and required [53].

Despite the obtained results, it is important to consider the limitations of this study. The search period was limited to the last 5 years. No documentary survey of methodological guidelines issued by HTA agencies was conducted. Not all actors involved and interested in HTA processes, such as representatives of manufacturers and patients, participated in discussions of research findings.

Recent publications have discussed HTA in MD with specific methodological approaches and objectives. Quintana and Cruz (2022) published the study discussing HTA models focusing on hospital acquisition of MD [54]. Ming et al. (2022) published the study discussing reports issued by HTA Agencies [37]. Freitas et al. (2023) published the study that explores the relevance of distinct value aspects for evaluating different types of medical devices according to stakeholders’ views [55]. Cangelosi (2023) published a systematic literature review to identify relevant studies and criteria used in medical device purchasing or procurement decisions [56].

Our research identified and characterized consolidated and innovative domains and methods to support HTA studies in MD. Through a systematic literature review, our findings can contribute to the field of incorporating MD in health systems.

Data Availability Statement

Search complete results can be made available upon request to the corresponding author.

Author Contributions

FT contributed in conceptualization, data curation, methodology, resources, writing–original draft and writing–review and editing to this work. DB contributed in conceptualization, data curation, methodology, resources, writing–original draft and writing–review and editing to this work. LT contributed in methodology, resources, writing–original draft and writing–review and editing to this work. MS contributed in review and editing to this work. ED contributed in validation to this work. All authors contributed to the article and approved the submitted version.

Funding

The author(s) declare financial support was received for the research, authorship, and/or publication of this article. This study was supported by the National Council for Scientific and Technological Development-CNPq, through the CNPq/DGITIS/SCTIE/MS No. 24/2021 public grant “Innovation in Methods and Application of Health Technology Assessment in Brazil.”

Conflict of Interest

The authors declare that they do not have any conflicts of interest.

Acknowledgments

We thank the National Council for Scientific and Technological Development–CNPq/Brazil and the Department of Management and Incorporation of Health Technologies (DGITS) for funding the two research scholarships for conducting this study. The study funder had no role in the study design, data collection, data analysis, data interpretation, or report writing. We would like to express our gratitude to Cintia de Freitas Oliveira, from the Institute of Health of São Paulo, for her contribution and methodological support throughout the study, as well as for the final review of the text. We also thank Dr. Evelinda MarronTrindade (Faculty of Medicine of the University of São Paulo), Telma Vinhas Cardoso (Faculty of Technology of Sorocaba), and Vania Cristina Canuto Santos (PAHO—Pan American Health Organization) for their valuable contributions in the final review. This study available at preprint SSRN: [55].

Supplementary Material

The Supplementary Material for this article can be found online at: https://www.ssph-journal.org/articles/10.3389/phrs.2024.1606343/full#supplementary-material

References