Validation of the McIntyre Audit Tool to measure haemodialysis nurse sensitive indicators

Background: Nurse sensitive indicators measure the quality of nursing care. Although there are some haemodialysis nurse sensitive indicators, there are currently no validated audit tools available to measure the indicators. Objectives: To test the validity of the McIntyre Audit Tool. Design: This study used a descriptive observation design conducted over two phases to assess face and content validity. Participants: An expert panel of haemodialysis nurses ( n = 13). Methods: Face validity (phase 1) involved 13 nurses in two focus groups who reviewed the audit tool with qualitative data generated analysed to identify common themes. Phase 2 used a modified version of the audit tool to test for content validity for each item and then scale level content validity was calculated by combining all item scores. Measurements: Ten nurses rated 26 indicators in the audit tool using a 4 ‐ point Likert scale to assess each item for clarity, relevance, appropriateness, and ambiguity. Results: All 26 ‐ haemodialysis nurse sensitive indicators achieved item content validity indices ranging from 0.825 to 1.00 with a scale content validity index average of 0.910. However, based on feedback from phase 2, 6 outcome indicators were removed from the audit tool to reduce staff burden and assist with ease of use. The final audit tool had an excellent average scale content validity index of 0.924. Conclusions: The McIntyre Audit Tool to measure 20 haemodialysis nurse sensitive indicators has been validated. It now requires feasibility and reliability testing before auditing the quality of haemodialysis nursing care


INTRODUCTION
Nurse sensitive indicators (NSI) are quality measures that capture those aspects of nursing care that directly impact upon patient outcomes (Afaneh et al., 2021;Krau, 2014). Quality indicators were originally developed by Donabedian in the 1960s to assess the structures, processes and outcomes of health care services (Best & Neuhauser, 2004). In the Donabedian (1966) framework, structures refer to the building blocks of care, including human and material resources, on which quality thrives (Cheng et al., 2014). Processes are the direct delivery of clinical care, and the outcomes are selfexplanatory with each aspect of the framework being interconnected (Rupp, 2018). Since that time, NSI have also been developed using this same framework.
Internationally, NSI are used to benchmark the standard of nursing care provided to patients (Afaneh et al., 2021). The initial NSI were first identified by Maas et al. (1996) which were related to ambulation, nutritional status and the effects of pain on the patient. These indicators aimed to establish causal links between the quality of patient care and the actual nursing care provided (Afaneh et al., 2021). Currently, NSI are used in many countries to measure nursing skill mix, hand hygiene compliance, rates of pressure injuries, incidence of falls and incidence of medication errors (Afaneh et al., 2021;Robertson, 2017). Indeed, a recent systematic review was able to identify the direct impact that the nursing skill mix had on 12 patient outcomes such as reduced length of hospital stay and lower mortality rates (Twigg et al., 2019). Globally, there is a shift within nursing to ensure the delivery of quality nursing care that also coincides with an increased level of patient expectation (Myers et al., 2017;Robertson, 2017). Understandably, the creation of NSI which are applicable to different areas of nursing practice have been necessary for some time (Mcintyre et al., 2019;Myers et al., 2017).

LITERATURE REVIEW
In 2013, the American Nephrology Nurses Association recommended developing nephrology NSI, and subsequently 10 indicators were identified (Thomas-Hawkins et al., 2017). These indicators included staff skill mix, the patient to registered nurse ratio, nursing hours per patient day, practice environment, patient centredness, care coordination, falls with injury, blood stream infections, 30-day rehospitalisation, and the patient's experience of care. At this time there was no haemodialysis specific NSI and many of the original 10 indicators were more suited to ward based nursing and did not fit in the context of haemodialysis nursing practice. Thomas-Hawkins et al. (2017) argued for a call to arms with an urgent need to identify further indicators suitable for haemodialysis nursing practice, and to test these indicators for validity, feasibility and reliability.
Recently, two studies have identified haemodialysis NSI. Gao et al. (2018), described 11 NSI which included the nurse-to-patient ratio and the incidence of fistula complications. However, Mcintyre et al., 2019) argued that some of these indicators (e.g., Kt/V compliance rate which assesses the adequacy of treatment and the ultrafiltration rate which controls fluid removal during a haemodialysis session) were not truly nurse sensitive as these were reliant on patient-related factors. In addition, Gao et al. (2018)

Design
This study used a descriptive observational design and comprised two phases with both phases using a peer review process with a group of expert haemodialysis nurses. Phase 1 examined the face validity of the McIntyre Audit Tool (MAT) and then phase 2 tested the content validity of the tool.

SAMPLE AND SETTING
The study was conducted across a large kidney health service in Brisbane, Australia which has three haemodialysis units. When assessing content validity of tools, an expert panel of 10−20 experts is recommended (Polit & Beck, 2006;Yusoff, 2019). The inclusion criteria for the expert panel were registered nurses working either full-time or part-time permanently in one of the haemodialysis units. Casual or enroled nurses were excluded from the study, as the aim was recruit only permanent senior nursing staff.

Description of MAT
The MAT comprises of a brief description of the purpose of the audit tool followed by three sections to assess face validity of the structural, processes, and outcome NSI. The intent was that structural and outcome indicators would be audited by nurse leaders of the haemodialysis unit, and process indicators would be audited by nurses working in the haemodialysis unit. Each of the sections contained the relevant indicators, some further information to explain what each indicator should be assessing, and then a response section (see Supporting Information: Table 1).

Data collection
Phase one involved two online semistructured focus groups which were conducted using Microsoft ® Teams. Data was collected through audio-recording for subsequent verbatim transcription for analysis.
Data was deidentified and stored on a secure drive.

Procedure
Phase 1 involved sending promotional flyers and emails to all nursing staff in the haemodialysis unit, and a brief PowerPoint ® presentation was given at routine staff meetings. Nurses who were interested were invited to contact the principal investigator if they had any questions or wanted further information about the study. Following ((iii) whether the rating system was understandable; and (iv) overall whether the content was appropriate for each indicator.

Data analysis
Testing face validity is a straightforward examination of whether the audit tool appeared to measure what is intended (Drost, 2011;Polit, 2015), and typically involves questioning experts (Heale & Twycross, 2015). Qualitative data generated during the focus groups were subjected to content analysis to identify common categories in short textual data (Elo & Kyngäs, 2008).

Data collection
For phase 2, a questionnaire was used to measure the content

Procedure
During phase 2, each of the participants received an email containing word-based copies of both the modified MAT, and an accompanying questionnaire. The participants where then asked to complete and return the questionnaire within 2 weeks. Reminder emails were sent to each participant towards the end of the first and second weeks.

Data analysis
Content validity was assessed by calculating the CVI of each item and then the entire scale (Almanasreh et al., 2018;Polit & Beck, 2006;Polit, 2015). Individual CVI (I-CVI) is the sum of the Likert scale scores. The total number of scores that have achieved 3 or 4 are added together and then divided by the total number of panel members (Almanasreh et al., 2018;Lynn, 1986;Polit, 2015). Scale-CVI (S-CVI/Ave) is calculated as a combined average of all I-CVI scores (Polit, 2015). I-CVI scores > 0.78 and S-CVI/Ave > 0.90 are considered acceptable for content validity of newly developed tools (Polit, 2015;Yusoff, 2019).

RESULTS
This study recruited 13 expert nurses for the face validity phase, and then 10 participants went onto complete the content validity phase.
The average clinical experience of the nurses was 20.3 years (standard deviation. 6.46, range 10-33; see Table 1).

PHASE 1 face validity
During the focus group sessions designed to determine face validity, 5-points to improve the MAT were identified by the expert panel.
First, the amount of information contained in the audit tool, which was designed to provide examples with current evidence to explain the NSI, was considered too long and descriptive (see Supporting Information: Table 2). Second, the scoring system (achieved/not achieved) was ambiguous as the panel members assumed that each example had to be achieved for each of the indicators listed rather than using the descriptors as an example of good nursing practice. At the end of phase 1, the audit tool was modified by the research team to include individual scoring for the definitions of quality care as well as 'not applicable' being added for each process indicators. Section A (structural indicators) was also modified by changing the heading from definition/meaning to rationale, and the first two indicators had never achieved added as a selectable option.
For section B (process indicators), it was identified that at least 50% of all items should be achieved. Face validity was now considered acceptable, and the modified audit tool was ready for content validity testing.

PHASE 2 content validity
Ten members of the expert panel completed content validity testing of the audit tool. Table 2 provides the I-CVI for each haemodialysis NSI. The four structural indicators achieved a very high level of agreement (all I-CVI ≥ 0.875) (see Supporting Information: Table 3).
The eight process indicators scored an I-CVI/Ave of 0.89 (see Supporting Information: Table 4). The 14 outcome indicators also achieved a good level of agreement with all scoring ≥ 0.80 (see Supporting Information: Table 5).
The MAT achieved an overall S-CVI/Ave of 0.910 (see Table 3) which is considered to be excellent (Polit et al., 2007) although based on the comments from the expert panel, 6 outcome indicators were removed (indicators 20−25; see Table 2) which further improved the S-CVI/Ave to 0.925.  The development of any clinical audit involves a five step cyclical process (Benjamin, 2008). First the problem needs to be clearly identified, and in this study, it was the need to measure the quality of haemodialysis nursing care. The second step is to establish what it is that you are trying to measure and perhaps identify the gold standard (Benjamin, 2008;Ullman et al., 2018). It will result in the development of an initial set of quality indicators, and in this case, 26 haemodialysis NSI were identified through a national consensus study using the Delphi technique (Mcintyre et al., 2019). The third step involves collecting data using an audit tool although this tool does need to be developed. In this study, we developed and tested the face and content validity of the MAT. The remaining two steps involves comparing or benchmarking the quality of nursing care provided either within an organisation or comparisons could be made with haemodialysis units in other organisations (Benjamin, 2008

IMPLICATIONS FOR CLINICAL PRACTICE
As nurses are responsible for providing most of the haemodialysis treatment directly to patients, it is crucial that the quality of their nursing care is assessed. Quality healthcare requires sufficient support structures, such as having the appropriate human and material resources to provide excellent care. The processes are the cornerstones of care and reflect the needs of safe practice and without these components then good patient outcomes will be difficult to achieve. This audit tool was developed to specifically assess haemodialysis NSI and provides a way to measure this information. The MAT addresses a known gap in clinical practice, and it can be used to monitor care at the local level or even to benchmark between organisations nationally or internationally.

CONCLUSION
The MAT has now been validated by a panel of expert nurses. This is the first step in creating a valid and useable audit tool that will enable haemodialysis NSI to be measured. Measuring specific indicators will not only make explicit the unique contribution made by nurses in haemodialysis units, but it will also support the identification of areas to improve. Further research is needed to test the feasibility of using the MAT.