Human Factors Quarterly Newsletter Fall 2015

Implementing Virtual Patient Simulation to Examine Diagnosis
Daniel Nystrom, MSAEEP Research Associate VA Center for Innovations in Quality, Effectiveness and Safety at the Michael E. DeBakey Veterans Affairs Medical Center, Houston, Texas, USA and Baylor College of Medicine, Department of Medicine – Health Services Research

Imagine you are a third-year resident physician completing your final rotation through the emergency department when a new patient arrives. Your job is to diagnose the patient and suggest a treatment plan to point the way for fellow clinicians to initiate treatment. How (or where) do you begin making your diagnosis? This is the setting and guiding principle I took when developing a virtual patient simulation (VPS) to study diagnosis while completing an Advanced Patient Safety Fellowship at VA National Center for Patient Safety (NCPS).

In general, virtual patient simulations (VPS) are interactive, computer-based clinical scenarios that are used to study or assess clinical reasoning. The emphasis of VPS on clinical reasoning makes it an ideal format for studying and training clinical diagnostic skills, and the online capability of most VPS software makes collecting large amounts of data a breeze. Despite these benefits, there is considerable difficulty involved with the development of VPS or other simulation scenarios.

One of the main difficulties in developing simulations that are tailored towards research is determining what features must be included to evoke the processes you are trying to study. In the case of developing the Diagnosis VPS, we needed to consider a number of features to elicit clinician diagnostic processes. To appropriately assess diagnostic skills with the use of VPS, we needed to: ensure the information pertinent to making diagnoses in actual clinical settings was available in the VPS, determine how the participants would interact with the VPS scenario, and identify how to interpret results that were collected from the VPS. Fortunately, the majority of these necessities were defined by a model of medical diagnosis.

In essence, the model of medical diagnosis used to develop the Diagnosis VPS describes where information can be obtained from the clinical work environment (e.g. computers, discussions with the patient, the appearance of the patient, etc.); what information is being obtained (e.g. the patient history, signs, symptoms, etc.); and how this information is transformed into cognitively useful representations of the situation to guide problem solving. In other words, the model of diagnosis describes how information from the environment contributes to the picture the clinician paints inside his or her head to describe and diagnose the patient. This model of diagnosis acted as a blueprint, identifying the important information that should be included in the simulation, defined the dynamics of the scenario, and provided us with a way to directly link data from the VPS to explanations provided by theories of reasoning and human factors psychology. After contemplating what the simulation would include and how it would look based on the model of diagnosis, we needed to find VPS software to construct and run our simulation scenario.

We chose to use the DecisionSim* software as the means to create and administer the Diagnosis VPS. The DecisionSim software provides a number of capabilities that allowed us to answer some interesting questions involving diagnosis (for more information on these capabilities, see the other article in this issue). In particular, DecisionSim enabled us to track the information clinicians selected while diagnosing the virtual patient and allowed us to administer questionnaires during the scenario to observe how the clinician’s thoughts about the patient evolved as a reaction to the information they selected throughout the scenario. Furthermore, since the VPS had direct links to the model of diagnosis, we were able to insert the choices made by clinicians during the VPS directly into the model. This process provided a rich, detailed picture of the way clinicians obtained information from the VPS environment and created a mental representation of the virtual patient’s current situation. We were also able to witness how the representations clinicians built guided the diagnostic process by informing clinicians where to collect further information to confirm or refute their current diagnostic hypothesis. Though these results are very exciting, they are still preliminary. Interested readers are advised to read more about the application of VPS as a method to study diagnosis in the upcoming Human Factors Ergonomics Society – Health Care Symposium 2015 Proceedings.

Currently, we are analyzing a larger data set to determine trends among a number of factors including diagnostic accuracy, clinician experience and specialty, the time it took to complete the scenario, response to scenario-based questionnaires, and the order in which information was selected while completing the VPS. After establishing the relationship between these variables, we should be able to determine the appropriateness of using VPS as a format to study and potentially train clinicians in the art of diagnosis.

Some of the studies we hope to complete in our future work by using the Diagnosis VPS platform include investigations on the influence of different aspects of clinical work on the accuracy and process of diagnosis. For example, we hope to study the effect of various information displays and interfaces (e.g. electronic health record displays, vital sign monitor displays, etc.) on the accuracy of diagnosis to determine the best way information can be accessed and presented to clinicians who are making a diagnosis. We also hope to begin research endeavors that evaluate the effect of different constraints of the clinical work environment (e.g. time pressure, cost constraints, etc.) on the process and timeliness of diagnosis.

In regards to research involving training and education, we plan to use the VPS format to familiarize clinicians with the process of seeking data to make a diagnosis and, as an indirect benefit, expose clinicians to a wide variety of diseases and disease presentations that are experienced in clinical practice. Assessing the impact of familiarity with data collection and exposure to a variety of diseases on clinician’s diagnostic strategies and accuracy will highlight the benefits of VPS as a method to train medical students and residents, and emphasize the possibility of VPS as a format for the perpetual training of diagnostic skills for experienced clinicians.

In conclusion, preliminary results on the feasibility of VPS as a format for studying diagnosis are encouraging, but there is quite a bit more research needed to validate the use of VPS as an efficient means of training clinicians in the art of diagnosis. It is our hope that thorough research investigating the use of VPS as a format to conduct research and education on diagnosis will provide evidence that VPS is a fruitful technique to solve questions about the diagnostic process, and improve clinicians’ ability to make accurate and timely diagnoses.

*Reference in this article to any specific commercial product, process, or service, or the use of any trade, firm or corporation name is for the information and convenience of the public, and does not constitute endorsement, recommendation, or favoring by the Department of Veterans Affairs.

Diagnostic Accuracy and Beyond With Decision Modeling Simulation Platform
Kira King, PhD, Director of Instructional Design, DecisionSim, Inc., and Jamie Owen, Instructional Systems Specialist, VA

In Daniel Nystrom’s article in this newsletter, he highlights the challenges with using Virtual Patient Simulation (VPS) to assess and educate clinicians on making accurate diagnoses. He then explains why he ultimately chose to use DecisionSim* software to administer the Diagnosis VPS. Nystrom describes how DecisionSim enabled his team to track information and ask learners questions about their thought process while diagnosing their virtual patient. As we work with a variety of clients, the data tracking capability of DecisionSim is proving to be one of its most impactful capabilities.

How DecisionSim Works

DecisionSim is a cloud-based simulation-learning platform. Its unique design features make it relatively easy to use, enabling designers (or authors) to customize their simulations, and to generate real-time data that assess how learners navigate their choices.

As learners, such as the clinicians in Nystrom’s study, go through a simulation, their choices branch down different paths to demonstrate the consequences of their decisions. This situates learners within the realistic context of a clinical story, empowers them to gather information and make decisions, and then demonstrates the consequences of their choices. Through this process, learners come to their own conclusions and insights as they discover what decisions worked well, and what actions led to poor or suboptimal outcomes. By situating learning within authentic contexts, we can improve the learning of VA employees, as well as the accuracy of the authors’ assessments. In turn, this contextual learning helps employees improve the high quality care they provide to Veterans.

As learners progress through a simulation by reviewing information and making decisions, their each and every click is recorded for future analysis. This enables authors to assess learner performance and look for correlations among a variety of factors.

For example, authors can compare and contrast learners’ diagnostic accuracy, confidence level, cost and test utilization, treatment, and disposition decisions—such as whether to admit or discharge a patient. The resulting data can include both numerical comparisons, as well as emergent themes related to the responses learners enter to open-ended questions. Organizations and divisions within VA can analyze data at an aggregate level to gather insights about learner performance, demographics, and overall perception of the simulation. These insights become even more impactful when they are tracked on longitudinal studies. These types of projects implement several simulations spaced over time, and may also include other data collection methods such as surveys, performance reviews, and other instructional activities.

Other factors could include investigations of cognitive bias, the effects of time pressure on decision-making, or an exploration of what criteria learners considered at various phases in the decision-making process.

Furthermore, simulations can be completed remotely when convenient for the learner. This is especially critical for health care providers who need to maximize patient time. Making learning more accessible and intuitive through the DecisionSim branching model can help to improve patient outcomes, as well as reduce medical errors and health care costs; all very important, especially in today’s increasingly complex health care environment.

As another example of diagnostic accuracy training, Denise Cochran, simulation center coordinator for the Malcom Randall VA Medical Center (VAMC) in Gainesville, Florida, and Linda Cowan, VA research health scientist, created a branched simulation to enhance nurses’ knowledge on how to stage and treat hospital-acquired pressure ulcers (HAPUs). The simulation begins with a short video from a virtual mentor who introduces learners to the setting of the simulation. Learners then watch a second video in which the mentor interviews a real VA patient. After watching the videos, learners stage a wound and determine treatment. If learners make a suboptimal or poor decision they are not immediately corrected; they receive feedback from the virtual mentor instead. This approach mirrors the real world where we cannot turn back the clock and reverse the poor treatment we chose. Instead, we have to fix problems we have created and try to improve the patient’s condition. Data from over 5,600 completed simulation sessions will be used to assess and enhance learners’ knowledge on how to stage and treat HAPUs.

Versatility of DecisionSim Beyond Diagnostic Accuracy Assessment/Training

DecisionSim can also be used to develop and enhance communication skills. For example, VA is creating a series of two DecisionSim simulations to help primary care physicians facilitate difficult conversations with terminally ill patients. Dr. Rosalyn Scott, Medical Director of the Simulation Center at the Dayton, Ohio VAMC, and a team of palliative care specialists, created these simulations to develop communication skills through a “choose your own adventure” storybook format. In these simulations, physician learners watch a short video of a physician/patient interaction that portrays the emotional impact of a terminal illness. After watching the video, learners decide how they want to guide the conversation by selecting one of several different statements or questions. Depending upon their choices, the conversation will either go well or poorly. Then, with the help of an online mentor, learners reflect upon the outcome and can go back and make a different selection for a different outcome.

The beauty of DecisionSim—and why it’s being successfully used within VA—lies in its simplicity and broad application to a wide range of learning needs. It can be used on its own or blended with other simulation approaches, as well as for skill-development in non-clinical domains. VA is even using DecisionSim for training administrators on patient scheduling, human resource negotiations, and for training physicians on the peer-review process.

VHA’s Employee Education System (EES) is constantly vetting innovative uses of DecisionSim. Manny Dominguez, Ph.D., EES deputy chief learning officer, plans to build it into the Virtual Medical Center or VMC; a first-of-its-kind virtual hospital campus and learning environment that will give patients and staff direct access to medical education, wellness tools, and hundreds of VA resources.

All this innovation with DecisionSim and other technologies shows how VA is leading the way in addressing human factors in health care and furthering the organization’s mission to be an established world leader in the application of clinical simulation-based strategies.

How do you know if DecisionSim is the right choice for a specific learning need?

If you’re interested in finding out if DecisionSim would be applicable to a certain need, help is a phone call or e-mail away. Staff can help you identify the features you should include within your simulation to create impactful insights. For example, a Help Center has both a planning document and design document you can use to articulate your simulation needs. Staff can also help you write engaging stories to support learning.

Creating realistic stories can be easier than you think. It’s often as simple as setting the stage, creating relatable characters and using simple media to create emotional engagement, such as short video clips, photographs with audio narration and text-based dialogue.

If you are interested in exploring DecisionSim in more detail, contact your Designated Learning Officer or the EES Learning Consultant for your division or Veterans Integrated Service Network (VISN)—and s/he will help you coordinate with EES to determine if DecisionSim is a good fit for your training needs.

*Reference in this article to any specific commercial product, process, or service, or the use of any trade, firm or corporation name is for the information and convenience of the public, and does not constitute endorsement, recommendation, or favoring by the Department of Veterans Affairs.

An Interview with Dr. Hardeep Singh
Christopher Petteys, MBA, Human Factors Engineering, Office of Informatics and Analytics

What is your role with VHA? What other institutions or organizations do you work for?

I am the Chief of Health Policy, Quality, and Informatics at the VA Health Services Research Center of Innovation, and Director of the Houston VA Patient Safety Center of Inquiry. I’m also an associate professor at Baylor College of Medicine and staff physician at the Michael E. DeBakey VA Medical Center in Houston.

How were you first introduced to human factors and what does that mean to you?

I have a strong interest in diagnostic error, which focuses on how clinicians err in making diagnostic decisions. While practicing full time in both private, rural and then academic primary care settings before becoming a researcher, I realized how my thinking was heavily influenced by everything around me, including the work environment. When I joined VA and started using the electronic health record (EHR), I became really interested in the way I was interacting with the system. This exposed me to some interesting human-computer interaction issues at the beginning of my career. I also realized that errors were a result of much more than just what’s in the doctor’s head and that our thinking and actions are strongly affected by the system around us.

Since then, lessons from informatics, human factors, and cognitive science have helped me understand more about diagnostic errors and the way the system positively or negatively influences diagnostic decision-making. For example, doctors are routinely held responsible for diagnostic errors, which could affect about 1 in 20 U.S. adults annually. One type of error is failure to follow up on critical information such as abnormal chest x-rays, which could lead to delayed diagnosis of lung cancer and poor patient outcomes. We started to look at details why test results are missed even when results are communicated to clinicians through the EHR, and here we found a human-factors approach very useful. In addition to information overload from too many alerts, we found reasons for missed results to involve interconnected human-computer interaction issues, policy issues, workflow issues, provider training issues, and technology issues. This multifaceted approach was foundational to our work, and I used human factors literature to help understand how to tie all this together.

How do you employ human factors principles in your work?

Our Safety Center was originally funded by the National Center for Patient Safety in 2007, and since then we have led a body of work in human factors and informatics issues related to test results communication in VA’s Computerized Patient Record System (CPRS). We apply human factors to three related areas: understanding and reducing misdiagnosis, improving follow up on abnormal test results; and ensuring safety and safe use of EHRs so we can use them to improve patient safety.

In all of this work, we’re trying to use a broad, multi-faceted sociotechnical approach where we’re thinking about not just the technology-related issues, such as software, hardware, and interface, but also process issues, organizational policies, and procedure-related issues. In one of our projects, a provider mentioned in an interview that despite being in the VA for eight years, he/she did not know that EHR notifications could be sorted, just like we can do on email. Another provider reported that certain types of alerts disappeared when you click on them and were “lost forever”. So these real-world investigations help us understand that when things go wrong, there are many reasons for it and we need to think broadly about many systems issues that affect us humans.

As researchers striving to understand complexities of health care, we have developed several conceptual frameworks. In most technology-related patient safety projects, we use an eight dimension sociotechnical model to help us think about implementation and evaluation issues, determine what we could be missing, what unintended consequences we should expect and what their potential solutions will be. The model has now been applied widely. It was used by The Joint Commission to do their analysis on EHR related patient safety issues and also by the Institute of Medicine in their report on Health IT and Patient Safety. For diagnosis we’ve developed the Safer Dx Framework (DX stands for diagnosis), which takes a broad social technical, human factors-oriented approach to measuring and improving diagnosis.

We used a sociotechnical approach to inform usability evaluation of a new clinical decision support tool we developed (AWARE) to help track missed test results. The extensive usability evaluation employed human factors principles to test beyond just providers interacting with the computer. Our team also used a sociotechnical approach to develop the Office of the National Coordinator for Health IT (ONC) SAFER Guides, which are risk assessment guides that institutions, organizations, and practices can use to become aware of recommended practices to mitigate certain high risk areas or situations. These guides were disseminated by ONC and are freely available on their website. We recommended practices for safe use of EHR applications such as computerized provider order-entry; clinical decision support; and facilitating use of EHR to reduce patient identification errors, communication errors, and other safety concerns.

Where do you see the greatest barrier to effective integration of human factors?

Translating science to action to make clinical impact is essential. There is still a lack of awareness of human factors principals and the benefits and value they offer. We need more people conducting human factors studies, but to my knowledge there are a fairly limited number of human factors experts in VA. A broader body of people is needed within VA who can be human factors champions and work on these issues in both research and operations. VA must also emerge as a leader in this field because it has so much potential. With an effective human factors workforce, we can raise the profile of this emerging field, and with a good research and operations partnership, we can overcome many barriers to improving care.

Where do you see the greatest potential for the application of human factors principles?

Now that VA is moving to a new EHR system, the enterprise Health Management Platform (eHMP), application of human factors principles and importing lessons from what’s being learned elsewhere is essential. What I mean is we need to build on lessons of others in developing, designing, implementing, and using these systems so that we don’t make the same mistakes that others have. I must add that CPRS was generally considered much better than the current commercial systems, so we will really need human factors to help us innovate beyond what’s known. Also as we encourage our Veterans to use more mobile tools, apps, and portals, it’s important to ensure we use technology correctly. The science of human factors can help us in predicting some of those unintended consequences, minimizing them, and if they occur, learning from them and figuring out how to minimize harm to end users. It’s important for building resilience.

Application of human factors will be useful for diagnostic error analysis and improvement. Studies we’ve done suggest that common diagnoses get missed in busy, chaotic primary care settings where there’s not enough time to spend with the patient and there is a burden from quality measurement, alerts and reminders in the EHRs. It’s not really surprising that diagnoses are missed in outpatient settings, including emergency rooms. We need to look at patient safety in the outpatient setting more closely and figure out new kinds of human factors-based strategies to improve the diagnostic process and reduce harm from diagnostic errors.

User-Centered Design: Narrowing the Gap between System Capabilities and User Needs
Brian Moon, M.Sc., Chief Technology Officer, Perigean Technologies LLC

Computerized clinical decision support (CDS) has and will continue to be an important feature of EHRs. However, the implementation of CDS has often fallen short of expectations. Lack of effective integration into clinical work is a chief complaint about adoption of CDS into clinical work.

In recent years, VHA has made a significant effort to ensure lessons learned about user experience in clinical practice are woven into CDS and EHR system design prior to deployment. Spurred by Meaningful Use certification requirements, “User-Centered Design (UCD)” has become a critical part of IT system development.

VHA’s Office of Informatics and Analytics, Health Informatics (OIA/HI) has provided VHA leadership a series of programs aimed at demonstrating UCD and developing lessons learned about the deployment of UCM methods and processes. This article describes several of these programs, which serve to inform how the gaps between system capabilities and user needs can be narrowed.

Describing macro cognitive health care work

The first program sought to explore the macro cognitive work undertaken across VHA – i.e., problem analysis in the UCD process (Figure 1). Modern health care can be described as a macro cognitive work system. In these systems people seek to adapt to complexity through functions such as sense making, coordinating, and re-planning. The ideal macro cognitive health care system would integrate technologies in ways that maximize the ability of people to adapt to complexity. It would do this through coordinating and synchronizing activities, and helping people anticipate surprise and error— all while preserving and enhancing the clinical experience between clinicians and patients. In other words, it would provide true clinical decision support.

Figure 1:  User Centered Design Process

Our team (which included Dr. Robert Hoffman from the Institute for Human and Machine Cognition and Mary Lacroix from Cognitive Medical Systems Inc.) conducted 60 interviews across seven VHA sites, exploring the macrocognitive work of doctors, nurses, physician’s assistants, and clinical support staff. Our interviews drew from methods of Cognitive Task Analysis to elicit a descriptive account of the cognitive functions and processes required to make decisions and perform complex tasks, individually and in teams. The resulting analysis produced diagrams and decision-requirements tables, organized around workflows that cut-across subdomains of health care practice: assessing, diagnosing, caring, ordering, managing (patients), and informatics.

From the analysis, we also developed a set of design seeds that should be considered in the design of CDS, EHRs, and other clinical support technologies. The design seeds are hypotheses about aiding performers around a domain-specific leverage point that are expected to usefully support performance. The seeds, organized around the workflows, included the following:

Validating CDS Guidelines

In early 2014, the office of Knowledge Based Systems, Health Informatics, Office of Informatics and Analytics at the Veterans Health Administration, drafted usability guidelines for CDS – based in part on the exploratory effort.

Figure 2. Simulated Formative and Summative Evaluation Prototype Screenshot

The Draft Guidelines address past concerns about CDS by providing guidance about the presentation of CDS functionality in user interfaces (UIs) based on knowledge of human cognitive work and decision making in naturalistic environments.

Figure 3. Simulated Formative and Summative Evaluation Prototype Screenshot

In order to validate the usability and utility of the Guidelines, our team attempted to follow them while implementing and evaluating five CDS User Interface Interventions (CDS UI Interventions) – i.e., conceptual design and planning through Summative Evaluation of the UCD process. We implemented the CDS UI Interventions in wireframes and mockups – key design decisions were intended to reflect the guidance provided by the Guidelines. We conducted iterative rounds of Formative Evaluations. The results of the evaluations were incorporated into high-fidelity prototypes, which we then assessed in a Summative Evaluation throughout. Examples of the prototypes are shown in Figures 2, 3 and 4.

Figure 4. Simulated Formative and Summative Evaluation Prototype Screenshot

Implications

Walking through the entire UCD process in the context of VHA revealed a number of challenges that are instructive for future efforts. The complexity of health care work should not be underestimated. While we discovered valuable insights during our exploratory effort, and while the guidelines help to ensure that cognitive requirements are factored into design, developing user-centered systems requires consideration of a complex set of issues. The value of UCD is that design hypotheses can be tested and eliminated if they do not account for the complexity.

Creating an evaluation context that is realistic and allows participants to exercise the designs is another key challenge. We spent a significant level of effort thinking through what data our evaluation participants would want to see just to be able to work through our simulated tasks. Executing a robust UCD process requires as much—and perhaps more—attention and effort be dedicated toward methodological issues as design.

The ultimate goal of our effort was to narrow the gaps between system capabilities and user needs. Yet it must be recognized that system constraints need to begin to be introduced very early in the design and development process. Programmers make a number of decisions to enable their work that, almost by definition, limit the capabilities of any given system. The iterative cycles between development and evaluation with the UCD process can help call into high relief where and how these decisions enable and hinder user performance.

Future applications

We are hopeful that the findings, analyses, and recommendations from our effort will help guide future UCD programs within VHA. With UCD being cited as a critical component of health care system development, our experience can provide insights about the UCD process and useful data for user-centered designs.

Video Lectures Offer a Deeper Dive into Clinical Decision Support from Human Factors Perspective
Rachel Wiebe, RD, CPHQ, Health Solutions Management, OIA

The Department of Veterans Affairs Employee Education System (EES) and the Health Informatics Initiative collaborated to produce the following video lectures on the topic of Clinical Decision Support (CDS) which are available through the VA Talent Management System (TMS) and accredited by Accreditation Council for Continuing Medical Education (ACCME), ACCME-Nurse Practitioner, American Nurses Credentialing Center (ANCC), and American Psychological Association (APA). VA has shared the lectures with all Federal partners, so they are available through the agency learning management system course catalogs at the Department of Defense, Indian Health Service, Health and Human Services and Centers for Disease Control and Prevention.

A General Introduction to the Art and Practice of Clinical Decision Support (TMS ID 9811) presented by Randolph A. Miller, MD, FACMI, provides an overview of CDS and of some of the theories and methods of human factors and Human Computer Interaction (HCI) as they relate to healthcare practice.

Ben Davoren, MD, PhD presents Clinical Decision Support: Emphasis on Users (TMS ID 9945), demonstrating the critical importance of user characteristics in the design, deployment, and utility of clinical decision support tools.

Clayton Curtis, MD, PhD presents Clinical Decision Support at the Point of Care: The View from 50,000 Feet (TMS ID 9931), which describes the challenge of scoping the terms “Clinical Decision Support” (CDS) and “Point of Care”. It also highlights the type of Clinical Decision Support being delivered in clinical software applications that health care professionals may encounter.

Algorithms in Clinical Decision Support (TMS 9928) presented by Peter Haug, MD, FACMI, introduces the algorithms and approaches used in this field as well as the software environment in which these kinds of tools can be most effectively implemented.