user-view

What is a summative test?

Summative testing is a comprehensive usability evaluation of a product or website in the context of how it is intended to be used in an actual use environment. Representative users take part in summative test sessions where they are asked to perform representative tasks under realistic conditions on final or near-final products or websites.

When should it be used?

Summative testing should be performed near the end of the product development phase on a final or near-final product or website. The goals of summative testing are to establish benchmarks for efficiency, effectiveness, and satisfaction; and to determine if the product or website has successfully met usability goals / success criteria that were established during formative testing.

What do you get?

The output from a summative test typically includes documentation of the findings for each task including performance summaries, a determination of whether or not usability goals / success criteria have been met for the product or website, a use error analysis discussing the results organized around a risk analysis of use, mitigation strategies for observed use errors, and recommendations for next generation releases or refreshes based on human factors principles.

 

What is a formative usability test?

Formative usability testing is an evaluation method that helps to “form” the product or website design. Representative users take part in exploratory test sessions where they are asked to perform representative tasks on actual and/or prototype products or websites.

 When should it be used?

Formative usability testing is typically performed iteratively during design and development phases to identify and resolve issues associated with the user interface design that impact efficiency, effectiveness, and satisfaction. Oftentimes usability issues can be identified as part of an initial formative test, design changes can be made to resolve or mitigate those issues, and the product or website can then be retested as part of a second formative test to ensure the usability issues have been addressed appropriately.

 What do you get?

The output from a formative usability test typically includes documentation of the findings for each task and recommendations for resolving or mitigating usability issues identified in the user interface design. Performance and satisfaction metrics can be used to establish success criteria for summative testing. Improvements are typically implemented quickly and further (iterative) testing is done to verify the usability of the improvements and to support continuous forward progress. Additional outputs from formative testing may vary based on the needs of the design and development teams.

What is a user survey?

An online user survey is similar to a structured interview with users but completed online and without an interviewer. A list of questions is displayed online and users’ responses are recorded.

 When should it be used?

Online user surveys can be used anytime during a product or website design and development lifecycle. They are a fast and affordable way to collect large amounts of data from a large number of participants. However, it is helpful to have very specific questions that need to be answered and you should be more interested in collecting the who and what such as demographics and quantifiable opinions, preferences, and/or behaviors as opposed to learning the why.

Results can be used to help inform product and website design based on users’ or groups of users’ responses. Online user surveys are used to determine demographic information to better detail the user population. They can also focus on users’ opinions and overall impressions or be used as benchmark studies.  For example, the same online user survey can track opinions over time by being launched every year.

 What do you get?

The output from a user survey typically includes a report that details:

• The primary goals of the user survey
• Questions asked to achieve those goals
• Findings in terms of answers to the survey questions, common themes, and trends
• Descriptive statistics as well as tabular and graphical displays to help clarify findings
• Recommendations based on findings

Don’t miss User-View’s poster presentation at the 2015 International Symposium on Human Factors and Ergonomics in Health Care: Improving the Outcomes; https://www.hfes.org//Web/HFESMeetings/2015HealthCareSymposium.html!

Jennifer Mauney, Caleb Furlough, and Janey Barnes  will be presenting.  Here is the summary:

Currently, many alert systems designed for EHRs are in a state which is negatively affect providers’ ability to prescribe appropriately and thus, is also affecting patient safety. There is a wealth of information in the human factors literature on the design and use of alerts but that information is either not known or has not been implemented in the design of EHRs. This paper will bring that literature to light, and demonstrate how to implement the recommendations to develop not only a better system for handling alerts in EHRs, but better CPOE (Computerized Provider Order Entry) as well. Serious usability issues with medication-based alerts displayed on EHRs were found after reviewing the data from the Meaningful Use 2 (MU2) Usability reports linked to the ONC CHPL website. The types of alert system issues found within the MU2 Usability reports and across various EHR vendors as well as human factors research that specifies how to deal with these issues will be described in this paper. In addition, recommendations for the design of medication-based alerts will be presented.

References

Belden, J., Patel, J., Lowrance, N., Plaisant, C., Koopman, R., Moore, J., Johnson, T. R., & Sonin, J. (2014) EHR Usability Style Guide. Retrieved from http://InspiredEHRs.com.

Belden, J. and Barnes, J. (2010) ARRA and EMR usability: What providers need to know.  Paper presented at the annual meeting of the Healthcare Information and Systems Society, Atlanta, GA.

Belden, J., Patel, J., Lowrance, N., Plaisant, C., Koopman, R., Moore, J., Johnson, T. R., & Sonin, J. (2014) Inspired EHRs: Designing for Clinicians. Retrieved from http://InspiredEHRs.com.

Bettman, J.R., Payne, J.W., and Staelin, R. (1986). Cognitive considerations in designing effective labels for presenting risk information. Journal of Public Policy and Marketing, 5, 1-28.

Furlough, C., Barnes, J., Mauney, J., Belk, A., Blanchard, L., Brooks, T., Brown, M., & Glasscock, N. (2014, 17-19 March). Observed usage errors during meaningful use stage 2 safety-enhanced design summative testing. HFES 2014 International Symposium on Human Factors and Ergonomics in Health Care Proceedings, Chicago, Illinois.

hguptamd. (2014, March 19). eprescribe [Video file]. Retrieved from https://www.youtube.com/watch?v=y16r7wtZ7S0.

Kesselheim, A. S., Cresswell, K., Phansalkar, S., Bates, D. W., & Sheikh, A. (2011). Clinical decision support systems could be modified to reduce ‘alert fatigue’while still minimizing the risk of litigation. Health Affairs, 30(12), 2310-2317.

Microsoft Health (2010) [image], retrieved from http://www.mscui.net/.

Miller, G. (1956). The magic number seven, plus or minus two: Some limits on our capacity for processing information. Psychology Review, 63, 81-97.

Osborne, P. D., Barsam, H. F., and Burgy., D. C. (1981). Human factors considerations for implementation of a “green board” concept in an existing “red/green” power plant control room. In R. C. Sugarman (Ed.), Proceedings of the 25th annual meeting of the Human Factors Society.  Santa Monica, CA: Human Factors.

Phansalkar, S., Edworthy, J., Hellier, E., Seger, D. L., Schedlbauer, A., Avery, A. J., & Bates, D. W. (2010). A review of human factors principles for the design and implementation of medication safety alerts in clinical information systems. Journal of the American Medical Informatics Association, 17(5), 493-501.

Pritchett, A. R., Vandor, B., & Edwards, K. (2002). Testing and implementing cockpit alerting systems. Reliability Engineering & System Safety, 75(2), 193-206.

Rasmussen, J., and Lind, M. (1981).  Coping with Complexity.  In H. G. Stassen (Ed.), First European Annual Conference on Human Decision Making and Manual Control. New York: Plenum Press.

Sanders, M. S. and McCormick, E. J. (1993). Human factors in engineering and design.  New York: McGraw-Hill, Inc.

Schneider, S. C. (1987). Information overload: Causes and consequences. Human Systems Management, 7(2), 143-153.

Sweller, J. (1988). Cognitive load during problem solving: Effects on learning.Cognitive science, 12(2), 257-285.

Wickens, C. D. (1984). Engineering psychology and human performance.  Columbus, Ohio: Merrill.

Wogalter, Michael S. & Kenneth R. Laughery (1996). WARNING! Sign and label effectiveness. Current Directions in Psychological Science, 5 (2), 33-37.

Wogalter, M., Desaulniers, D., and Godfrey, s. (1985). Perceived effectiveness of environmental warnings.  Proceedings of the Human Factors Society 29th annual meeting.  Santa Monica, CA: Human Factors Society, 664-668.

Young, S.L., (1991). Increasing the noticeability warnings: effects of pictoiral, color, signal icon and border. Proceedings of the Human Factors Society 35th Annual Meeting, pp.580-584. Santa Monica, CA.

When someone says, “I am a teacher” or “I am an accountant,” folks get it. People know what those jobs are.

But when I say, “I am a human factors specialist,” I might as well say I am an alien from Mars—except even then folks would still have a better idea of exactly what I was.

In explaining what a human factors specialist is, I try to start with something most people already know: “Have you heard of the term ergonomics before? Like when an office worker gets his or her computer, desk and chair set up to prevent carpal tunnel syndrome?”

That’s a job people are familiar with. So I build on their knowledge, explaining that ergonomics is a sub-specialty of human factors. If you think of ergonomics as designing products to work with people from the neck down, then think of human factors as designing products to work with people from the neck up—the thinking part.

That’s what I do. I am a human factors specialist and I work with companies to assure them that their products, services and websites work for people.

On the human factors side (the cognitive side) I make sure:

• Folks can easily find a phone number in the cell phone.
• Medical devices are safe and easy to use so doctors and nurses can think about patients instead of trying to figure out how a medical device works.
• The ATM gives you back your ATM card before giving you your money. Because once you get your money, nothing else matters and you will drive away from the ATM.

On the ergonomic side (the physical side) I make sure:

• The back hatch of the SUV can be easily opened even when someone is wearing gloves
• The radio and heating / air conditioning controls are within reach of the driver even when the driver is wearing a seatbelt.
• Medication packaging takes into account decreased hand and finger strength of patients, so that patients can open the packaging

My explanation drives some of my colleagues crazy. Among experts the terms human factors and ergonomics are frequently used interchangeably. But when I separate the physical versus the cognitive aspects of human factors, folks understand what it is I do.

And making things easier to understand is also part of what I do.

Here is a list of Usability, User Centered Design, User Experience, Human Factors talks I’ve found so far.  I remember my first years at HIMSS when there was nothing related to Usability, User Centered Design, User Experience, Human Factors…we are growing in usability maturity.

April 12, 2015 —

8:00 AM – 4:30 PM CT – This session is a poster session and will only be available to attendees of the Physicians’ IT Symposium

Summative Scenario-Based EHR Usability Testing in a Community Hospital –

David Schlossman, MD, PhD, FACP, MMI, CPHIMS

 

 8:00 AM – 4:30 PM CT – This session is a poster session and will only be available to attendees of the Nursing Informatics Symposium

Patient Story: Designed for Telling

Ruth Schleyer, MSN, RN BC

Maribeth Stein

 

 8:00 AM – 4:30 PM CTEMR Implementation in a 9 ED Hospital System – This session is a poster session and will only be available to attendees of the Nursing Informatics Symposium

Alexander Bautista, RN, BS

Rhonda Staton, RN, BSN

 

 8:00 AM – 4:30 PM CT

Patient Story: Designed for Telling – This session is a poster session and will only be available to attendees of the Nursing Informatics Symposium

Ruth Schleyer, MSN, RN BC

Maribeth Stein

 

April 13, 2015 —

10:00 AM – 10:30 AM CT

HIMSS Spot | North Hall | B Lobby

HIMSS Spotlight: Updated Usability Maturity Model Launch

 

10:30 AM – 11:15 AM CT

HIT User Experience Community Meetup

 HIMSS Spot | North Hall | B Lobby

 

2:30 PM – 3:30 PM CT

Improving EHR Usability through Awareness Computing

Wayne Fellmeth, MD, FAAP

Room S404

 

2:30 PM – 3:30 PM CT

Data Warehousing with Semantic Ontologies

Richard Biehl

Room S100A

 

5:00 PM – 6:00 PM CT

HIT User Experience Community Reception

Booth 4891 | Health IT Value Suite

 

April 14, 2015 —

7:30 AM – 8:15 AM CT

American College of Cardiology (ACC) EHR Usability Meeting

Room W191 | McCormick Place Convention Center

 

10:00 AM – 11:00 AM CT

Data-Driven Decision Making for Individual and Public Health

Shannon Pohl, RN, BSN, MS

Fred Rachman, MD

Room N427

 

  12:15 PM – 12:45 PM CT

Turning the Tide: User Experience and HIT (Pt 1)

Andrew Gettinger, MD, FCCM, FCCP

UX Thought Leaders from Allscripts, athenahealth & Epic

 Booth 4891 | Health IT Value Suite

 

 1:00 PM – 1:30 PM CT

 Turning the Tide: User Experience and HIT (Pt 2)

Constance M. Johnson, PhD, MS, RN, FAAN

UX Thought Leaders from Allscripts, athenahealth & Epic

 Booth 4891 | Health IT Value Suite

 

April 15, 2015 —

10:00 AM – 11:00 AM CT

Proudly Accepting Patient-Generated Health Data – This session is part of the special program called, Patient Engagement Forum.

Mark Groshek, MD

Carolyn Kerrigan, MD

Room S401

 

11:30 AM – 12:30 PM CT

Switching EHRs: A National Survey of Physician Users

Kenneth Adler, MD, MMM

Robert Edsall, MA, MPhil

Room W196A

 

1:00 PM – 2:00 PM CT

 You Completed Meaningful Use Summative Testing…Now What?

Janey Barnes, PhD

Greg Wolverton, FHIMSS

Room W196A

 

2:30 PM – 3:30 PM CT

Solving Vexing Design Problems for Nurses: eMARs and BCMA

Nancy Staggers

Lorraine Chapman

Room W196A

 

April 16, 2015 —

10:30 AM – 11:30 AM CT

Incorporating User-Centered Design into Quality Improvement

Jason Cadwallader, MD

Justin Morea, DO

Room S103

 

 12:00 PM – 1:00 PM CT

The Embedded Designer: Better Usability through Observation

Deborah Chan, MASc, PEng

Carleene Banez

 Room W194

 

 

Today, Thursday, November 13th is World Usability Day.  This year’s team is Engagement: www.worldusabilityday.org. Be sure and check out events online and in your neighborhood.  If you are online check out World Usability Day Challenges from the HIMSS Health IT Usability Committee! If you are in my neighborhood, check out the TriUXPA Event at SAS Campus in Cary, NC! What can you do today to improve the engagement we have with technology products and services?

HIMSS (http://www.himssconference.org/education/education-sessions) and the Human Factors and Ergonomics Society in Health Care Conference (https://www.hfes.org//Web/HFESMeetings/2015hcs-ccit.html)  are done for 2015.

Great talks.  Networking.  Inspiration.

While we seem to be having the same discussions as we did in 2014 and in 2013 and in 2012 and there is much work to be done to improve the usability of healthcare IT…

We are also talking about getting the focus on the patient.

We are also talking healthcare IT being more than an EMR.

And we are also talking about interoperability.

Are we making progress on the usability of healthcare IT?

At one of the meetings, the question was asked; has there been any benefit from the Safety-Enhanced Design Meaningful Use 2 program.  The response; yes.

The program has shined a light on the manner in which teams apply User Centered Design process:

1) Terry Fairbanks and Raj Ratwani. Human Factors Perspective on Advancing EHR Usability & Safety.  Health IT Implementation, Usability and Safety Workgroup October 10, 2014. http://www.healthit.gov/facas/calendar/2014/10/10/policy-hit-implementation-usability-safety-workgroup.

The process has shined a light on the manner in which teams carryout and report Summative Usability Methods:

2)  Christine Buchanan, Anthony Threatt, Matthew B. Weinger, & Anne Miller, Vanderbilt U. Medical Center, High Variability in Summative Usability Test Methods & Reporting Among Clinical Informatics Vendors Complying With Federal Certification Requirements.  Presented at International Annual meeting of Human Factors and Ergonomics October 2014 3.  http://www.hfes.org/web/HFESMeetings/2014HFESAnnualMeetingProgram.pdf.

The process has allowed at least one organization to report out the frequency of identifying critical usage errors that might harm patients using only a summative test vs using a full User Centered Design process + a traditional Solution Development Lifecycle:

3)  Gary Gartner.  Improving the Safety of HIT with a User-Centered design Process. Poster presented at the International Symposium of Human Factors and Ergonomics in Healthcare June 2014 3.  http://www.hfes.org/web/HFESMeetings/HCSPresentations/HCS2014Gartner.pdf

The process has allowed at least one organization to report the ranking of prioritized features (e.g., eMAR, Reconciliation, etc.)  based on findings reported on the ONC CHPL website for 9 EHR products.  The rankings take into account task failures x error analysis score (frequency x severity):

4) Caleb Furlough, Janey Barnes, Jennifer Mauney, Alisha Belk, Laura Blanchard, Teri Brooks, Megan Brown, Naomi Glasscock, Merryl Gross, Ellie Hunt, and Hasmik Mehranian. Observed Usage Errors during Meaningful Use Stage 2 Safety-Enhanced Design Summative Testing. Proceedings of the International Symposium of Human Factors and Ergonomics in Healthcare June 2014 3: 81-86, doi:10.1177/2327857914031012. http://hcs.sagepub.com/content/3/1/81.full.pdf+html

 

I attended my first annual HIMSS Conference in 2005. I initially attended to network with other human factors professionals and was surprised that I could not find others discussing how human factors methods were being employed to improve health IT. How things have changed!

When I first got involved with HIMSS, usability was not part of the conversation. I remember I could not find anyone who was talking about how to impact the usability of a Health IT products or process. This year at HIMSS16 (February 23 – March 4) the usability conversation is prevalent and maturing as evidenced by the sessions focused on the broader context of human factors and usability applied to applications and processes. Don’t miss:

Education Sessions

• UX in HIT: A History of How We Got To Where We Are Today
• UX in HIT Development: Vendor Perspectives
• Identifying Solutions for Nurses’ HIT Pain Points
• UX Makes All the Difference for Clinicians
• UX, ROI, and the ‘Bottom Line’
• What Does the Evidence Tell Us About UX?
• Creating a Shared Vision:  The Roadmap for Successful User-Centered HIT
• UX Design: Five Steps For Designing “Healthy” Clinical Apps
• Optimizing Drug-Dose Checking to Minimize Alert Fatigue
• Usability Assessment of an Opioid Clinical Decision Support Tool
• Pilot Implementation of Tablet Devices on Medical Resident Rounds
• The Impact of Usability Issues on Non-Clinical Hospital-Based EHR Users

And if you are still at the conference on Friday at 10:30 – 11:30, Come to Lando 4201 for my presentation with Celeste Mayer and Jennifer Mauney

Shared Responsibility in Improving Alarm Safety

I can’t wait to connect with the many colleagues I have worked with (actually volunteered with) from the Usability Task Force; now the HIT Usability Community. Read how involvement in the HIMSS Usability Task Force provided me great opportunities on the Meet Our Members section of the HIMSS website. Usability birds will be flocking together. Come check out HIMSS usability and get involved!