Human Factors Quarterly Newsletter Spring 2016

Day in the life of a VA Clinician: a field-based perspective from Primary Care, Specialty and Emergency Department Physicians
Kurt Ruark, MBA, Program Manager, Emerging Health Technologies, Human Factors Engineering; Danielle Hoover, MD, MPH, Primary Care Physician, VA Ann Arbor Healthcare System; Douglas McKee, MD, Primary Care Physician, Viera Outpatient Clinic; Edward Posnak, MD, Dermatologist, Orlando VA Medical Center; and Francine Sandrow, MD, Emergency Physician, Philadelphia VA Medical Center

Background: The Emerging Health Technology (EHT) service from the Office of Informatics and Analytics’ (OIA) Human Factors Engineering (HFE) program assesses recent technology advancements for impact at the point of care within the Veteran Health Administration (VHA). To identify the realistic value and to define the user experience (UX) evolutionary path, it is imperative that the current difficulties be acknowledged. EHT initiated a study to determine common challenges facing clinicians in their present-day environments to help inform the strategic direction of VHA’s Health Information Management Systems.

Approach: For a true representation of VA clinician experience, the Day in the Life assessment is in a third-person narrative format, and describes ambulatory care clinicians’ interaction with patients, staff, the physical environment, and the EHR. The issues and problems described in the story are inspired by true events reported in user-feedback sessions with Primary Care, Specialty, and Emergency Department field clinicians in an outpatient setting. To control scope, inpatient, surgical and long-term care environments were excluded from the evaluation. The assessment gathered published evidence related to the characterization and extent of identified challenges. Current well-performing Computerized Patient Record System (CPRS) functionalities were also noted, such as the ability to: see remote data on a patient from any VA, follow the fill history of medications prescribed to patients, and see the results of a specialty consult immediately after the specialist signed their note. Finally, EHT put forward a conceptual context of a future-state in the Primary Care setting with many of the issues resolved or greatly reduced, while depicting clinicians’ feedback for how such a clinical utopia would improve patient care and the Veteran and employee experience.

Introduction: “What happens on a normal day in the life of a VA clinician?” This question might engender an acute anxiety attack for clinicians as they think of the overwhelming volume of tasks they must accomplish (and what they’ll forget before finishing their notes). However, it also serves as a reminder that administrators, developers, engineers, and system architects truly don’t know the answer to this question. The goal was a user-centered problem analysis, validated by a different set of clinicians, with input collected from VA clinical leadership (a bottom-up and top-down approach).

Clinicians who engage in VA patient care day-in and day-out form workflows that accept limitations, fight to evolve work-arounds that deal with minutiae, or fly away to other job opportunities. In fact, VHA is losing approximately 15% of new hires annually, with about 20.7% of Medical Officers (clinicians) quitting within two years (Workforce Management and Consulting Office, 2014). Although VA is actively hiring more staff, 64% of the clinicians who left VA were “regrettable losses” (i.e. not lost to retirement and chose to leave VA), and a striking ~98% of them were resignations and not transfers to other agencies (Department of Veterans Affairs, 2015). Even more disturbing, is that only 1.9% of clinicians in training see a government or military practice setting as a desirable career choice, likely attributable to the perception that VHA is largely bureaucratic, and lead by administrators instead of clinicians (The Medicus Firm, 2014). Hiring more clinicians focuses on the symptom of retention and doesn’t address the problem with the employee experience. Only changes to the workplace environment can support improved morale.

Burdensome, non-clinician-centered workflows in the EHR push literally hundreds of mandated alerts (individual computer messages) to the clinician per day. Many of these interrupt the workflow or clinician’s train of thought, and are considered by clinicians to have low clinical significance to patient care (Shan, et al., 2006) while taking time away from that care. Many of the clinicians engaged also spoke about frequent inefficiency issues (e.g. extra clicks, taxing documentation requirements and redundant workflows) that force more engagement with the computer than with the Veteran, evidenced in all of the assessment’s targeted care settings: Emergency Department (Hill, Sears, & Melanson, 2013), Primary Care (Street, et al., 2014), and Subspecialty (Lanham, Sittig, Leykum, Parchman, Pugh, & McDaniel, 2014). Clinicians were most worried about the patient safety concerns, such as the example of having no “back” button when processing alerts in CPRS leading to a lost report for a patient with a new lung mass on a CT scan.

The persistent and overwhelming volume of tasks related to both EHR workflow and general clinic flow presents challenges not only to the VA clinician employees’ experience and job satisfaction, but ultimately to the Veteran’s experience and overall quality of care provided in the VA health care system. The continuity and longevity of this type of care delivery system depends on effective, efficient, and engaging interactions between clinicians and patients, a relationship that must become a cornerstone for advancing the health and well-being of the enterprise. The main themes identified in the assessment that were experienced by clinicians, along with some interesting facts surrounding them, are listed below.

Findings:

1) Lack of support staff

Although Primary Care has an established staffing model, full commitment to staffing has yet to be reached, with nearly 33% of Patient Aligned Care Teams (PACTs) not meeting the minimum recommended ratio, and one in 10 positions left vacant (VSSC, 2016). Specialty services are exploring staffing models, each with different levels of maturity, but with no standard panel-size established, access can be significantly inhibited, forcing profoundly short appointment times and leading to Veterans’ perception of rushed care.

2) Inadequate dedicated time allowed outside of face-to-face visits

The general clinician schedule includes 7.5 hours of scheduled patients per day. When new time is needed, say an unscheduled walk-in or phone call addressing an urgent concern, the total clinic time is not adjusted; such ad-hoc additions have become the norm. This obviously eats into, or completely consumes, the lunch and break periods while, more often than not, creeping into after-clinic hours.

3) Interruptions

Many clinicians experience a vast array of cognitive interruptions during the clinic day, originating from the information system (alerts, emails, and instant messages) or in the physical environment (knocks on door, pages, and phone calls). Many interruptions can be attributed to the prior issues identified, with the lack of dedicated time or support staff to address them. When there is no accommodation for walk-ins in schedule blocks, the provider can quickly get overwhelmed by the sheer numbers of patients needing to be seen, leading to long waits for all of the patients and unhappiness for the providers. This total lack of control over their clinic schedule was described as a common reason for clinicians wanting to leave VA employment. While not the focus of the assessment, variation in clinical policy was noted, as well as a broad range of interpretations for the same national policies.

4) Disparate data systems and difficulty navigating data

Clinicians are information gatherers and decision makers, with searching and acquiring data consuming vast amounts of time. While the physical examination is an important data gathering task, information in the Veteran record can be very influential in the development of the plan of care as well. Looking for a key piece of information in a Non-VA hospitalization record, buried within an unsearchable image or faxed paper report, takes precious time. A Veteran’s individual care history can be vast in size and not integrated with all VA/Department of Defense data in a quickly searchable format. Current keyword-only search results are commonly so slow that clinicians intentionally avoid them, creating a moral dilemma when information may be available, but not accessible. This leads to delays in care, duplicate orders, time removed from the more critical decision-making process, and the potential for significant safety issues, such as missing important abnormal findings in imaging or labs.

5) Medication management

A critically important task that can lead to better health, although driven entirely by the validity of the data available, medication reconciliation is a time-intensive process that forces a clinician to collect and merge information, not just analyze and review. Forcing a clinician to perform data-entry of medications from non-VA sources can take many minutes depending on clinical complexity, averaging five minutes in some studies (Plaisant, Wu, Hettinger, Powsner, & Shneiderman, 2015). Pain management, using opioid medications and other controlled substances, makes up 9.4% of all VA outpatient medications. These Veterans can require extended time to support, often complicating not just their encounter but those of adjacent scheduled Veterans. When frustrated with their care regarding narcotic pain medications, Veterans may threaten clinic staff to a point where staff safety becomes a concern as well.

6) Caring for Veteran’s multiple needs during a visit

Whether in primary care, a specialty clinic, or even the Emergency Department, Veterans get so little time with clinicians that they are desperate for support when one is with them. Many Veterans have considerable clinical complexity with frequent mental health and chronic pain conditions, and often more than one clinical complaint (reason for visit). Many times Veterans ask for help from the individuals in front of them, rather than the “responsible” clinician. “Responsible” enterprise systems would be designed with the need for such flexibility in mind, making Veterans’ needs the top priority while integrating other clinical requirements, such as methods to communicate quickly between clinicians, sharing patient requests for needed support, and facilitating Veteran education for patient responsibilities. Delaying one Veteran’s care to support another isn’t Veteran-centric. Each Veteran has important and unique needs, and putting Veterans first in a Veteran-centric care delivery system would entail each appointment being tailored to their individual needs, not a one-size-fits-all clinic schedule.

7) Lack of understanding of end-to-end clinical functions by policy and IT decision makers

A common theme expressed by clinicians and evidenced by recent scandals is a tendency to focus on measures and isolate interest on issues with national attention. Problems and issues are being addressed one at a time, with new policies and additional mandated tasks, while the total impact to clinical workflows is not being considered, nor are the unintended consequences readily recognized. Clinicians bear the brunt of this, with policy and process changes made to help improve measures frequently shifting the work or data entry tasks to clinicians. When the end-to-end experience is understood, it is easy to see why VA is not the employer of choice for clinicians and why Veterans are critical of their experience within VHA. Recognition of the strengths and weaknesses within the VA care delivery network is vital to continue improvement and ensure the best care and services are provided to Veterans.

Human Factors at the Front Lines: CPRS Use and Clinical Workflow Efficiency
Alan Calvitti, PhD, Veterans Medical Research Foundation

This is the first in a series of reports featured in the VHA Human Factors Quarterly Newsletter that tell the story of how the VA Computerized Patient Record System (CPRS) impacts primary care provider workflows. This report summarizes key findings and methods from an observational time and motion study called the Patient Centered Electronic Health Record (PACE) study of office follow-up visits conducted at four San Diego VAMC primary care clinics.

Report #1 - Summary of clinical and EHR activity patterns

An outstanding problem with current generation Electronic Health Record (EHR) designs (including CPRS, Epic, etc.) is they don’t integrate well with clinical workflow. Ideally EHRs should provide users fast and intelligent interfaces and automation for common, high-level clinical tasks (e.g., patient information retrieval, documentation, order entry, care team coordination), enabling the clinician to focus on the patient, taking history, thinking, and making and documenting clinical decisions during the limited time-window of visits. Instead, EHR workflows force clinicians into low-level manual tasks like navigating across screens (e.g. 12 CPRS Tabs), scrolling through lengthy notes to mentally gather fragmented patient information, and stepping through nested pull-down menus to order medications, labs, imaging consults, and even return-to-clinic appointments. Further, structured and unstructured information are not linked, so doctors manually perform double work by documenting their order entry activity in notes. Much of this activity happens while patients and doctors carry on verbal communication. We emphasize that in primary care, unlike specialty care, the “reason for visit” typically spans a broad range of conditions and problems, including chronic and acute conditions, pain management, mental health, and social factors.

For this study, unobtrusive, wall-mounted video recording was used to capture visit room clinical activity, focusing on doctors’ visual attention to computer and patient, and patient-doctor communication. Usability software (Techsmith Morae) was used to capture time-stamped CPRS activity (screen and mouse clickstream). After extensive human coding of these signals (computer-assisted ethnography), we merged these data layers along their common timeline. Hierarchical and sequential pattern analyses were used to quantify common patterns and variations across 111 visits by 21 primary care physicians, where each saw between two and six distinct patients. These results were given context by clinician interviews, and led us to identify candidate areas of usability improvement of CPRS. Since the main study findings have not yet been published, we will omit technical details to focus on the relevance to system design (e.g., goals, functions, tasks, decisions).

One of the key findings of our study, Figure 1, emerges from summative activity based on the video review and gaze data. In the aggregate, doctors spend more time looking to the computer (42% of visit time) than looking at the patient (35% of visit time). The remainder of a median 29-minute visit is spent in physical exam and other activities such as tending to artifacts and paper records, answering phone or pager, or communicating with nurse or other staff. The right column shows total CPRS mouse clicks aggregated to CPRS functions. Most clicks occur in the Notes tab (40%) followed by Orders (25%), and notably, ~10% each in Labs and Meds which are used primarily for information retrieval. (Based on a separate time-motion analysis, the time spent in Notes is even higher, closer to 50% of total time on EHR. This distinction is important as time is a universal measure of activity, whereas mouse click counts are not relevant with touch or voice interfaces).

Figure 1. Key Study findings (click to expand)

Another important finding is that doctors tend to cluster in behavior. Some (e.g. doctor 09, top) tend to have shorter visits (e.g., <20min) and use the EHR less during the visit (50min) and use CPRS much more (~500 clicks) in the visit. Total click-count per visit is positively correlated with both visit length (6 min for every additional 100 clicks) and with gaze dominance ratio (attention to EHR by attention to patient), though we can’t say which is the cause and effect. Although the scatter is not shown here, clustering can be based on comparison of interquartile and range extremes of visit length and click count. Many doctors have told us in interviews and group discussions that they shift some EHR work (e.g. prep or review notes, order entry) before or after a visit. Thus doctors have some leeway in their workflow during the visit, but the strategy is doctor dependent. Interestingly, the proportion of gaze-to-patient does not change significantly across doctors (i.e. consistently at least 10 minutes, even for extreme doctors). However, both gaze-to-patient time and EHR tasks are highly fragmented during visits. This will become clearer after conducting detailed task switching analysis and visualization. In sum, some doctors tend to extend their visits, and much of that time is filled with additional work primarily using the Notes and Order activities in CPRS. Three of the doctors (03, 10, 12) used DragonSpeak voice dictation during visits, but this does not seem to save mouse clicks, probably because many clicks are associated with navigation and menu selection tasks that voice control does not affect. In subsequent reports, we will describe navigation patterns and detailed order entry activity, as well as the CPRS components that doctors share with patients.

In summary, the PACE study provides baseline evidence for real-world CPRS usage and clinical workflow patterns, albeit with small samples, at a single site and focusing on primary care with data capture only during the window of the visit. An ongoing follow-up study called QUICK expands to a larger sample of primary and specialty care at two sites (including University of California at San Diego (UCSD) which uses Epic’s EHR and adds (masked) keyboard clickstream capture, cognitive load instruments like (modified) NASA TLX survey and eye tracking, again in real clinical settings. On the UCSD side, we also performed Epic audit-log analysis to study EHR work performed before and after visits by the entire care team. The similarity results between the comparable aspects of PACE (summative) and QUICK (formative) suggests that top level findings are stable and likely replicable. Further, each additional layer of data joined can enrich the type of tasks and activities investigated in relation to clinical workflow and human factors and suggest how next generation EHRs should be designed to measure and scale up this type of analysis (e.g. capture client-server traffic to profile EHR use). However both studies are primarily limited to questions of efficiency rather than patient safety or other important aspects of care.

Ambivalent Compliance Versus True Belief
Linda C Williams, BSN, MSI, CHSE, VA National Center for Patient Safety

It is a truth that the Universal Protocol as acknowledged by a single person in possession of surgical and procedural skills, will be in want of a pre-procedure time-out.

- With apologies to Jane Austin

We know just how critical it is to have patient safety-oriented policy and procedures in health care settings. But how do we take the essence of safety expressed as words on a page and make it a sustained reality of practice, moving past ambivalent compliance to actual engagement?

Knowing that humans are typically less capable in the midst of noise and chaos, high fidelity simulation (HFS) is crafted to hone what Itiel Dror calls “error trapping” skills, by marrying a brief pause-in-time with active risk-detection. HFS scenarios demonstrate the value of taking a pause specifically for error detection; a time-out prior to first cut or invasive insertion. The concept of a sterile cockpit, borrowed from aviation, makes it acceptable in medicine to create protected time, a step-back for a safety check. It is in this protected time and space that skills of error-detection grow. The opportunity to practice in simulation improves the ability to detect and mitigate potential errors in real life, which is often chaotic.

Our work in high fidelity simulation has been focused on improving teamwork and communication. Frustration has grown seeing unenthusiastic participation in these time-outs, accompanied by a gradual decrease in the use of pre-procedure briefings. This occurs, even as the data show that making use of the time-out prevents inadvertent harm to patients. The key to reaching the fully engaged, true-believer stage of participation is successfully discovering potential harm, thereby finding a new value in a time-out during the universal protocol and recognizing it as the last barrier protecting a patient from accidental harm. This experience can be gained through simulation or “error trapping.” Dror declares it to be a skill that can be taught and honed with practice. He proposes that strategies for learning to detect errors be taught early in medical school and continually honed.

With this in mind, and knowing that it is easier to identify errors in others’ thinking and behavior, we build obvious errors into beginning-level HFS scenarios. We ask in debriefing sessions that participants identify contributing factors – rather than focusing on individual performance. The move from seeing the opportunity to blame a colleague (or the creators of the simulation scenario) to identifying problematic system or device design is typically a big first insight.

System-design issues from which to create HFS scenarios are, unfortunately, plentiful: noise and distractions; device design flaws (e.g. a stapler-cutter that may cut without stapling); invisible things such as oxygen trapped under a drape creating a fire hazard, or infection risk from aerosolized Mycobacterium; a monitor that has had a recent software update that results in a reversal of the image; an x-ray or other studies inserted from another patient’s Electronic Health Record (EHR); misinterpreted color-coding; fatigue from having to maintain an uncomfortable, stressful posture in order to use a device at an odd angle; suboptimal lighting sources or available electrical supply; missing information without a clear resolution.

In the high-fidelity simulation, a few questions direct the procedure team’s attention. These or others may be appropriate depending on the patient and the procedure:

• What could go wrong, and what do we need in order to be prepared to mitigate it?
• What confirmation do we have of correct patient and correct site?
• If the laparoscopic procedure goes unpredictably awry, who is on-call for surgery?
• Is there a chest tube handy if we inadvertently collapse the lung?
• Will the use of oxygen and cautery create fire risk for this procedure?
• What possibilities do we need to anticipate?
• Will the patient need blood? Is it available?
• Does the implant match the patient?
• Is all equipment available and ready for use? o Are there additional images available?

At the Intersection of Human Computer Interaction, Human Factors, and Informatics: An Interview with April Savoy, PhD
Christopher Petteys, MBA, Human Factors Engineering, Office of Informatics and Analytics

Chris: What is your role with the Veterans Health Administration (VHA)? What other institutions or organizations do you work for?

April: I am a health services researcher and the Director of the Human Computer Interaction and Simulation lab at the Richard L. Roudebush VA Medical Center (VAMC), in Indianapolis, Indiana. I’m also an assistant professor and director of informatics at a regional campus of Indiana University. After receiving my PhD, I worked as a program manager and research associate at SA Technologies Inc., where I designed and evaluated situation awareness-oriented user interfaces for military combat and systems.

C: How were you introduced to the concept of Human Factors?

A: With my background in computer science, I knew of human computer interaction (HCI). Until I started my doctoral studies at Purdue in Industrial Engineering, I thought I also knew something about human factors. The university offered programs in HCI and Human Factors. In the Human Factors course, I saw the explicit need for human abilities, limitations, and mental models to influence actual design. Prior, when I wanted to improve design, I always went back to the system. I’d focus on improving hardware and software to tweak system capabilities, but I never reached out to the users or considered the human limitations in information processing. In that particular course the light bulb went off and I realized that I should talk to those who are using the system and consider how humans are processing information and their mental models. Through that graduate school program, I was introduced and reintroduced to the concept of human factors and learned to apply that knowledge to simulations and designs to investigate impacts on end users and their workflows.

C: How do you use/employ Human Factors in your work?

A: Human factors is now integrated into everything I do, both inside and outside VA. My whole philosophy is to create designs and health information technologies (HIT) that are user-centered. I don’t want to just make screens prettier; I want to make them more useful. I think using that as a foundational mantra helps me to align with the needs and goals of improving HIT at VA. My underlying theory is to look at HIT at the intersection of care coordination and situational awareness, so we can enhance Veteran care throughout the system by improving communication, information displays, and critical decision making. User Centered Design process is something I implement into all projects. A lot of people talk about usability testing, but addressing usability concerns starts well before usability testing. My research agenda can be summed up into three phases: 1) In the first phase we develop cognitive and user requirements by finding out from the users what is needed to help them do their work. 2) The second phase includes designing prototypes with effective displays that adhere to situation awareness-oriented design principles. 3) The final phase is evaluating the organizational and workflow impact of the HIT for that individual or team.

One of the current projects that applies these theories is a major consults project. At the heart of the project is improving communication between Primary Care Providers (PCP) and specialists. The goal is to recommend design enhancements to improve communication and information sharing throughout the care delivery process. Another project is related to the remote patient monitoring program collaborative between sleep and telehealth services, and integrating feedback data into Continuous Positive Airway Pressure (CPAP) machines to improve care delivery. A third project is looking at the mobile needs in medical facilities to see what mobile applications are needed to improve their workflows, and ultimately patient health outcomes.

Much of this work is done in our Human Computer Interaction and Simulation Lab which integrates human factors engineering with clinical information systems to improve care for Veterans. This physical lab affords an environment that enables variables to be controlled, which facilitates observing direct impacts of the interested interventions. If interested in working with the lab, users can contact April.Savoy@va.gov.

C: Where do you see the greatest barrier to effective integration of HF principles?

A: From my perspective, it’s critical to find an effective means of communication and a streamlined process to strengthen collaborations between research and operations. A fragile point for the successful integration of human factors is the translation of research findings to user requirements for development and operations in a timely and effective manner. Regarding timing, researchers have commented on the difficulties caused by operations timelines that are shorter than those of research. I have seen quality improvement projects implemented that seem to address some of the difficulties. Regarding effective communication, I have witnessed situations where people talk right past each other, because someone is using terminology the others don’t understand – or – someone is using the same exact terminology that means something different to others, which can impede progress. Having avenues for effective communication and access to leaders that have experience on both sides are crucial for relaying a common framework and ideology. To help address translation issues, we utilize more visualizations. So, as soon as we begin to understand what others are requesting, we develop frameworks or conceptual models to help illustrate the actual concepts. This enables people to easily react to miscommunications and adjust to correct the points made. This helps people to avoid working in silos and connect better which results in them becoming more familiar with working with each other to build effective technologies.

C: Where do you see the greatest potential for the application of HF principles in VHA?

A: I am very excited about Connected Care and the possibilities it entails. I think introducing the philosophy of the “internet of things” is a great potential for VA to increase care coordination, access to information, and communication between facilities. Improving the connectivity between providers, patients, and caregivers is a great potential area for applying human factors in VHA. Everyone has a cell phone, tablet, computer, etc., but it is not the presence of information technology that creates great advancements, it is the design of applications that are user-centered and address major health concerns of Veterans and Clinicians regardless of location. We can develop technologies that can help Veterans track their health care journey and communicate issues directly to their physicians. This allows rural Veterans to avoid the long drive to see medical professionals. So I think the development of medical technologies to support a connected structure of mobile health and connected care is the greatest potential for the application of human factors. We have a great opportunity to introduce the user centered design process in an early stage. This is ideal because we don’t have to adjust at the end and start redesigning, we can start now and design applications in an optimal way, the first time.

Highlights from the 2016 International Symposium on Human Factors and Ergonomics in Health Care
Rachel Wiebe, RD, CPHQ, Office of Technology and Information Strategy

The Symposium was held April 13–16, 2016, in San Diego, California, and featured cutting-edge presentations, posters, and workshops on emerging issues in healthcare and human factors. In this Spotlight we feature two presentations from the symposium our readers might want to learn more about.

Mind the Gap: Team Training in Health Care presented by Sarah Henrickson Parker, PhD of the Virginia Tech Carilion School of Medicine and Research Institute along with Lillian Su, MD, and Mary Patterson, MD, of Children's National Medical Center, highlighted a series of studies using Team Strategies and Tools to Enhance Performance and Patient Safety (TeamSTEPPS). Dr. Su described her team’s work in assessing gaps in clinical practice through cardiac simulations and use of Intensive Care Unit (ICU) video to examine delays in starting cardiac resuscitation. Her team is now improving training and teamwork to decrease delays. The session also focused on the role of listening in trauma situations, particularly in terms of frequency and timing of different types of communications. Their work found the nature of communications changes throughout a trauma case and seems to be cyclical.

Along the lines of TeamSTEPPS, VA’s National Center for Patient Safety offers Clinical Team Training which we encourage you to explore!

Patient-Centered Non-Routine Events (NRE) in Ambulatory Care presented by Shilo Anders, PhD, of the Center for Research & Innovation in Systems Safety at Vanderbilt University reported on a study that sought to understand the patient’s view of health care that is suboptimal, low quality or unsafe and asked the question: Can patients report events that are potential patient safety risks and different from that of their clinicians?

Dr. Anders and her colleagues collected information about non-routine events (NREs), defined as anything that occurs for a patient that is less than optimal during their care or visit from patients/family members and their clinicians. What they found is patients reported substantively different NREs than their clinicians and reported NREs that had patient safety implications. In addition, anesthesiologists and trained patient partners in the research categorized the NREs and found that patients and clinicians categorized NREs differently. This work lays the groundwork to tailor interventions in clinical areas and to study linkage of patient-reported NREs to other outcome measures. For more information, check out this slide presentation!