AGRASS Questionnaire: Assessment of Risk Management in Health Care

ABSTRACT OBJECTIVE This study aims to assess the development and the validity analysis of the Assessment of Risk Management in Health Care Questionnaire (AGRASS). METHODS This is a validation study of a measurement instrument following the stages: 1) Development of conceptual model and items; 2) Formal multidisciplinary assessment; 3) Nominal group for validity analysis with national specialists; 4) Development of software and national pilot study in 62 Brazilian hospitals 5) Delphi for validity analysis with the users of the questionnaire. In stages 3 and 5, the items were judged based on face validity, content validity, and utility and viability, by a 1-7 Likert scale (cut-off point: median < 6). Accuracy and reliability of the questionnaire were analyzed with the Confirmatory Factor Analysis and the Cronbach’s alpha. RESULTS The initial version of the instrument (98 items) was adapted during stages 1 to 3 for the final version with 40 items, which were considered relevant, of adequate content, useful, and viable. The instrument has 2 dimensions and 9 subdimensions, and the items have closed-ended questions (yes or no). The software for the automatic collection and analysis generates indicators, tables, and automatic graphs for the assessed institution and aggregated data. The adjustment indices confirmed a bi-dimensional model composed of structure and process (X2/gl = 1.070, RMSEA ≤ 0.05 = 0.847, TLI = 0.972), with high reliability for the AGRASS Questionnaire (α = 0.94) and process dimension (α = 0.93), and adequate for the structural dimension (α = 0.70). CONCLUSIONS The AGRASS Questionnaire is a potentially useful instrument for the surveillance and monitoring of the risk management and patient safety in health services.


INTRODUCTION
The quality of health care is on the global health agenda, and patient safety is one of its critical components 1 . In order to improve the quality of health care by increasing the patient safety, health care systems are suggested to implement risk management practices 2-4 and quality management/improvement methods applied to the health care safety [5][6][7] .
The risk management in health care is highlighted as one of the seven stages towards patient safety in the National Patient Safety Agency in the UK 2 . The National Quality Forum of the United States also includes identifying and reducing risks and hazards as the fourth of its 34 evidence-based safe practices. In the Brazilian health policy context, risk management in health facilities is the first objective of the Programa Nacional de Segurança do Paciente (PNSP -National Patient Safety Program) 4 .
The relationship between patient safety and quality of care occurs in two approaches: in the first one, patient safety is considered a dimension of the quality of health care 8,9 ; while in the second one, safety is considered an aspect of health care regardless of their quality 10 . Overall, the World Health Organization 6 (WHO) includes quality improvement methods as one of the 11 key topics of the multi-professional patient safety curriculum guide. In Brazil, the quality management focused on risk reduction is present in the regulation of health care 7,11 .
Considering the regulatory framework of the Brazilian health care, sanitary surveillance professionals are properly supported to oversee the implementation of quality management 11 and risk management in health care 12 . The two management models can be used to improve safety. The external and internal sanitary inspectors and accountants need to have a wide and flexible insight to identify the implementation of principles and the achievement of objectives of the patient safety management, and not only a registry judgment of one model or another.
Many risk management and quality management models have been suggested for organizations in general 13 , developing many variations to health care. However, few studies devoted themselves to validating instruments to assess the implementation of quality 14 and risks 15 management activities, showing the need of the area, especially in Brazil, which has no validated instrument for its local reality.
In a partnership between the Agência Nacional de Vigilância Sanitária (ANVISA -National Sanitary Surveillance Agency), the Pan American Health Organization (PAHO), and two universities, a combined model was suggested based on the principles of risk management and the quality of patient safety 14 . Based on this model, the Assessment of Risk Management in Health Care (AGRASS) was established, aiming to coordinate external sanitary audits and inspections performed in the Brazilian context and assisting the self-assessment of health care. Considering this background, this study aims to assess the development and the validity analysis of the AGRASS Questionnaire.

METHOD
This is a validation study with a quality and quantity approach. The study was developed in partnership between ANVISA, PAHO, the Universidade Federal do Rio Grande do Norte (UFRN) in Brazil, and the Universidad de Murcia in Spain. This article describes the development and validation of the AGRASS Questionnaire in five stages, as shown in Figure 1.

Stage 1 -Development of the Conceptual Model and Items
An original and combined conceptual model was developed to show the key structures and processes of patient safety management in health care. This model was based on: wide insight of the literature on national and international technical documents; risk and quality management; models to improve the quality and risk management in health care [16][17][18][19] ; benchmarking risk management publications in European health care 20 ; international recommendations for patient safety practices that included risk management 2,3 ; and Brazilian sanitary legislation 4,11,12 . This stage was performed in 2016 and had the initial dimension consensus by two doctoral researchers, who are specialists in management and improvement of quality in health care, and a third professional, who is a quality manager in a hospital certified by the National Accreditation Organization (NAO). The details of the reviewed references and of the conceptual model are available in a previous publication 13 .

Stage 2 -Proper Assessment
The basic version of the AGRASS Questionnaire was properly reviewed by a group of six experts. The method of analysis was qualitative, by detailed discussion of the questionnaire items, aiming to adjust it to the best systems of patient safety management and considering the Brazilian context. The method should also be concise and consistent with the requirements of the sanitary surveillance. For the saturation analysis and consensus, three meetings were needed, developing the second version of the AGRASS Questionnaire in July 2017.

Stage 3 -Validity Analysis with National Sphere Specialists
The second version was globally analyzed and reviewed item per item by ten ANVISA experts in August 2017. For the consensus, the nominal group technique 21 was used, with three voting moments (two of them were face-to-face and the other one by distance). The questionnaire items were judged according to four criteria related to face validity, content validity, utility, and feasibility. The criteria were formulated in the form of closed-ended questions on a Likert scale from 1 to 7, on which 1 was "totally disagree" and 7 "totally agree". The questions were the following: "Is the item relevant to the risk management in health care?," "Is the item related to the dimension it wants to measure?," "Is the item feasible in the context of risk management in health care?," and "Is the feedback information useful to detect improvement opportunities in the risk management in care health?." Each expert could also include comments regarding each item. This stage was useful for removing items, adapting terms, and adding fundamental item clarifications, developing the third version of the AGRASS Questionnaire.

Stage 4 -Development of Software and National Pilot Study
With the product of the previous stage, the Laboratory of Technological Innovation in Health of UFRN developed the AGRASS System, which consists of two modules -mobile and web.
The mobile module provides the questionnaire to be answered in health units, stores them, and sends the inserted responses to the web module. The web module automatically analyzes the data and provides individual or aggregate reports (sets of health care by region, state, municipality, and other groupings of institutions) with the report of the assessed health care and the indicators for the implementation of risk management, the latter showed in Table, Radar chart, and Pareto chart. The degree of risk management implementation in the health care assessed can be described according to individual items and group items. The items allow a positive response and are all equally weighted for the total estimate of risk management implementation in health care. For the descriptive analyses of group items, the percentage of positive responses is estimated, namely, the total positive responses in the group items regarding all items of the AGRASS Questionnaire or all items of the dimensions and subdimensions.
In possession of the electronic format of the AGRASS Questionnaire, a distance learning course was offered for 120 state and municipal professionals of the sanitary surveillance of all Brazilian federation, between September and December of 2017, based on the ad hoc didactic material that contains the AGRASS conceptual model. With the help and supervision of four researchers, the trained professionals applied the AGRASS Questionnaire in health care, which established the pilot study and the assessment of the offered course. The sample size was defined in a stratified manner and proportional to the amount of hospitals with intensive care unit (ICU) in each health care system of Brazil. The professionals made their choice conveniently seeking the largest services of their locality. Sixty-two hospitals participated in the pilot study. All of them had an ICU, because this type of service is considered a priority for sanitary surveillance, within the framework of the Plano Integrado para a Gestão Sanitária da Segurança do Paciente (Integrated Plan for Patient Safety Sanitary Management) 22 .

Stage 5 -Validity Analysis with Instrument Users
Finishing the validation stages, the 74 professionals who applied the instrument were invited to participate in a validity analysis using the Delphi technique 21 . The examination was performed with an electronic validation form sent by e-mail, which included questions about the AGRASS Questionnaire using the same guidelines, criteria, and cut-off points of the validity analysis of stage 3. This examination occurred in February 2018, within 23 days, during which up to three reminders were sent to the non-respondents (10, 15, and 20 days after the examination).

Data analysis
The validity analysis of the questionnaire items had quantitative and qualitative components. The data analysis in stages 3 and 5 considered the median of each of the four criteria. The items that obtained a median equal to or greater than 6 in all criteria were included in the final version of the AGRASS Questionnaire. The items that had some criterion with a median lower than 6 were discussed by the respective groups, and may have undergone adjustments, being submitted to a retrial by the experts. If the item had any medians lower than 6 in the second vote, it was removed. As for the qualitative component, the comments recorded in the assessment form and in the discussion after the presentation of the results of each vote were considered, even if the criterion had reached the cut-off point.
For the reliability analysis, internal consistency was estimated using the Cronbach's alpha (Appendix 1) for the complete AGRASS Questionnaire. The structure and process dimensions were estimated using the questionnaire responses of the pilot study. The reference values considered for a good adjustment were χ 2 /gl < 3.0; RMSEA < 0.05; TLI > 0.95; CC ≥ 70, and VME ≥ 50. For the RMSEA, in an ideal situation, the lower value of the 90% confidence of interval (90%CI) includes or approaches zero or is not greater than 0.05; and the higher value is not very large, namely, it is lower than 0.08 23 (Appendix 2).

Ethical aspects
This study was approved by the ethics committee in local research, with the opinion number 75662517.2.0000.5292, and it followed the requirements established for its realization.

Conceptual Model
The conceptual model developed in the first stage of the AGRASS Questionnaire regards the set of structures and processes that aim to constantly improve patient safety ( Figure 2). The dimension called "firm structure" consists of five subdimensions: awareness, accountability, ability, action, and safety culture. Additionally, the dimension called "key processes" consists of four subdimensions: risk identification, risk analysis and assessment, risk treatment , and risk communication. The processes joined by arrows represent the natural flow of integration between them.
The conceptual model aimed to present the practices of risk management in health care in a playful way and, for this, used an analogy with the art of a juggler (Figure 2). In this figure, the juggler represents the person responsible for the patients' safety in health care, as well as all other professionals involved. The balls balanced in the air represent the risk management processes, which must be fully present and integrated with each other. The firm soil and pleasant temperature, contextual factors that help the artist's work, show the importance of the presence of leadership structures and systems and favorable organizational psychology regarding climate and safety culture.

Proper Assessment and Validity Reviews
The first version of the AGRASS Questionnaire developed in stage 1 included 98 items regarding the dimensions of the conceptual model and was submitted to proper assessment in stage 2. After the analysis by six experts, 50 items were removed and a second version of the AGRASS Questionnaire with 48 items was developed.
The second version was submitted, in stage 3, to the validity analysis by 10 ANVISA professionals, whose aspects are shown in Table 1. After the first voting round, 12 items did not reach the cut-off point and underwent discussion before a second vote. After the second voting round, only two items obtained a median lower than 6 and were therefore removed. Three other items were removed, even reaching medians greater than or equal to 6, because of the agreement among the experts' group on whether they were disposable questions. Four items merged into a single item and 26 of them underwent adjustments. Chart 1 shows the results of this stage.
The third version of the questionnaire had 40 items. An important contribution of this stage was the addition of the definition of each dimension and the descriptions about the items, to make the questionnaire clearer and more reliable. This version was used to develop the AGRASS System. Of the 120 sanitary surveillance technicians who enrolled in the course "Inspection of good risk management practices in health services," 74 were approved and invited to participate in the second phase of the content validity analysis. Of these, 32 technicians signed an informed consent form and answered the validation study questionnaire. The participants' description is shown in Table 1. The group had members from all Brazilian regions (2 from the Midwest, 1 from the North, 7 from the Northeast, 17 from the Southeast, and 5 from the South). This group approved the relevance, usefulness, feasibility, and relationship with risk management of all 40 AGRASS Questionnaire items in the first vote, according to the results shown in Chart 1.
The assessment measures of adjustments were used to verify the adequacy of the model to the data. It showed that the instrument is valid and appropriate to the two dimensions of structure and process, with < 3 chi-square ratio and degrees of freedom (X2/gl = 1.070). The RMSEA criterion reinforced the indication of good adjustment of the tested model  The standardized model shows that all items -except item 9, "The institution promoted its professionals training in the area (risk management, quality management, patient safety, etc.)" -presented significant factor loads (p < 0.05). However, as item 9 did not compromise the dimension model in general, it was kept in the questionnaire because of its attributed content validity by the experts.
Regarding the reliability analysis using Cronbach's alpha, the AGRASS Questionnaire (40 items) had a result of 0.935 (high consistency), the structure dimension (12 items)

DISCUSSION
This study has a potential contribution to the improvement of surveillance and management of health care, as it provides a valid instrument for the assessment and monitoring of the organizational conditions that ensure the provision of health care insurance. After five stages of validation, the AGRASS Questionnaire was considered relevant, valid, useful, and feasible to measure the implementation of risk management in Brazilian health care. The questionnaire can be used to: assess health care regarding the implementation of risk management practices, guiding the planning of improvement interventions; review health care to measure improvement after performing interventions to implement risk management practices; and compare individual or aggregated health care (e.g. region, municipality, or state) regarding the level of risk management practices implementation. The instrument can be applied by external evaluators (sanitary inspectors, certifiers, accreditations and auditors, among others) or in self-assessment initiatives of health care systems. Accountability Definition of an organizational system responsible for the risk management assistance and accountability. 3

Contributions to Patient Safety
Ability Necessary resource provision to accomplish the risk management. Includes human resources in appropriate quantity and qualification, time for professionals to devote to activities of risk management, financial resources, equipment, and inputs.

4
Promoting patient safety culture Periodic assessment, feedback from employee assessments, and intervention in identified weaknesses, to constantly improve the patient safety culture.

Risk identification
Risk identification shows the manageable risks. Risk identification may be: retrospective, which identifies previous incidents in the institution and their probability of repetition; in real-time, which identifies risks at the critical time when a support incident may occur; and prospective, which identifies risks based on the future possibility of incidents, although they have not yet occurred at the institution.

13
Risk assessment and analysis Risk analysis and assessment activities aim to better understand safety problems before implementing risk reduction actions.

4
Risk treatment Planning activity and actions to improve patient safety, therefore avoiding, reducing, or transferring the risks. 3

Risk communication
Constantly communicating the activities of risk management and the risks to managers, health professionals, patients, and external regulatory bodies.

Process Integration
Risk management processes should be performed in a combined method, with or without the use of integration techniques (e.g. 5 whys, FMEA, etc.).

Total 40
The AGRASS Questionnaire meets an international and national need. Information systems are one of the six main strategic interventions that WHO stimulates for the quality of health care 9 and, particularly in the WHO Patient Safety Program, safety measurement is a priority objective 24 . However, even well-developed countries in patient safety, like England, have been urged to improve patient safety information collection 5 . This improvement opportunity was also evident in an assessment of government decisions regarding patient safety in countries of the Organization for Economic Co-operation and Development (OECD), which showed that assessment and monitoring are not very advanced 25 .
In Brazil, although the PNSP assessment and monitoring were planned 26 , there is a wide scope for improvement in this aspect. The Sistema de Notificações para a Vigilância Sanitária (NOTIVISA -Sanitary Surveillance Notification System) and the annual assessment of patient safety practices implemented by ANVISA are important, but insufficient 26,27 . Thus, the AGRASS Questionnaire presents itself as an alternative instrument to produce information for decision-making in patients' safety based on 40 items, which can be comparable to 38 simple indicators and 2 composite indicators (items 15 and 30). This new instrument can be used to measure the patient safety at the level on which it has a greater lack of assessment, which is its organizational component. The quality of care should be monitored from a systemic perspective, including the technical level of individual care, but also the organizational levels of the establishment of health and the health care 18 . Patient safety indicators have been proposed for all levels 28,29 , but few are based on organizational criteria. The measurement of safety should be complete, not only based on adverse results or events 28,30 . Measuring the quality of care is not easy and requires balance.

The Validity of the AGRASS Questionnaire
The stages of experts' validation enabled us to reach the objectives of developing and validating the content of the AGRASS Questionnaire. In addition to the extensive review of the literature, the final items were approved regarding the face validity, which is the logical and clear importance for risk management; content validity, which is the ability of each item to measure the expressed concept of each dimension and subdimension of the questionnaire; and the construct validity, which measures how much the items adjust to the dimensional structure of the questionnaire. Content validation seeks to value opinion makers and recognizes the experts' important contribution 23 . Regarding reliability, although it had acceptable internal consistency, it is emphasized that it depends on the evaluator's training and on the correct application of the instrument, demanding and assessing the necessary vouchers. In addition, composite reliability values reinforce this aspect.
The name AGRASS Questionnaire emphasizes the risk management in health care, but it should be understood as an instrument for assessing patient safety management based on risk management and quality models. Regarding the size of the structure, this study highlights the structuring of leadership systems for patient safety and an environment defined by a favorable organizational culture. On the size of the risk management processes, those were included (identification, analysis and assessment, treatment, and communication) appear directly or indirectly in most risk and quality management models 13,16,28 . Risk identification subdimension includes retrospective methods (such as indicators), in real-time methods (such as checklists), and prospective methods (such as the safety walk-around) 3,13 . Another issue emphasized by the model is that processes should not be independent, but integrated with each other, showed in the subdimension "integration of processes." The fact that the AGRASS Questionnaire was originally developed for Brazilian health care favors its validity in the local context. Other instruments with similar objectives were developed abroad to evaluate the management of clinical risks 15 and quality management 14 but were not transculturally adapted to Brazil. Although these instruments seem similar to the AGRASS at first, they have important differences in their conceptual model and items.