Six Evaluation Components

Our team began by breaking down the evaluation into six main components:

1—Define Scope

We kicked off the project by facilitating a meeting with the entire procurement team. This included clinicians, engineers, technicians, and PMs. Together we defined the project scope: 

• Users: ICU nurses, telemetry nurses, anesthesiologists
• Workflows: ICU monitoring, low acuity monitoring, monitoring during surgery
• Environments: ICU, telemetry unit, operating room (OR)
• Devices: central station monitor, high / medium / low acuity bedside monitors, transport monitor, telemetry pack

2—Develop Scenarios

In order to identify tasks to include in the evaluation, my team shadowed clinicians throughout the hospital, and observed how they interacted with patient monitors. In addition, we conducted user interviews where we asked clinicians what tasks were essential to their work with patient monitors. We identified 5 main scenarios.

3—Recruit Participants

A total of 28 participants were recruited for the evaluation. Participants were a mix of ICU nurses, telemetry nurses, and anesthesiologists from various units across the hospital, and used patient monitors at least once per week.

4—Prepare for Testing

Three main test environments were identified: operating room, ICU, telemetry unit. The testing labs were setup to resemble these environments. In addition, ambient noise was played in the background to add to realism.

5—Evaluate the Device

Each test session had one participant and included orientation, training, usability testing, debrief interviews, and questionnaires. My colleague and I alternated being the facilitator and note taker. The facilitator acted as a nurse and was in the same room as the participant while the note taker observed remotely from another room. To reduce recency bias, the order that systems were tested in was counterbalanced.

Both quantitative and qualitative data relating to usability was collected.

6—Analyze and Report

Following testing, we synthesized data from all 28 sessions. Quantitative data was used to calculate usability scores for each system, and qualitative data was used to understand the severity of the scores as it relates to patient safety.

Findings and recommendations were shared with the procurement team through a 1-hour presentation and 130-page report.

2020 HFES Symposium Poster

Specific findings from the evaluation are under and NDA and thus cannot be disclosed. However, my team prepared a poster for the 2020 HFES Symposium which compared usability findings from this study to a similar study conducted by the HHF team in 2009. The poster highlights usability issues that were common in both studies.

Click here to see the PDF version. I'd love to get in touch to discuss our testing methodology further!

Reflections & Learnings

As my first time conducting a comparative usability study of this scale. I made many mistakes, however, these mistakes made me a better test facilitator, allowing me to successfully execute two subsequent procurement studies.

Some of my learnings are as follows:

• Do not ask leading questions when facilitating. I often asked questions that were too leading, biasing participants into agreeing with me instead of sharing their thought process. I learned that asking open ended questions often elicits root causes that I might not have thought of; reducing confirmation bias.
• Analyze data as you go. Our team didn't start analyzing majority of the data until after all 28 sessions were complete. This made it hard to remember session details, particularly if there were gaps in the notes.
• Limit the scope to what is necessary. Our team felt the need to include as many monitoring tasks as possible resulting in some sessions lasting 3 hours. Moving forward, a better understanding of device usage would allow us to prioritize tasks based on criticality to patient safety and clinician workflows.