blog


What is Summative or Validation Usability Testing?


What is a summative test?

Summative testing is a comprehensive usability evaluation of a product or website in the context of how it is intended to be used in an actual use environment. Representative users take part in summative test sessions where they are asked to perform representative tasks under realistic conditions on final or near-final products or websites.

When should it be used?

Summative testing should be performed near the end of the product development phase on a final or near-final product or website. The goals of summative testing are to establish benchmarks for efficiency, effectiveness, and satisfaction; and to determine if the product or website has successfully met usability goals / success criteria that were established during formative testing.

What do you get?

The output from a summative test typically includes documentation of the findings for each task including performance summaries, a determination of whether or not usability goals / success criteria have been met for the product or website, a use error analysis discussing the results organized around a risk analysis of use, mitigation strategies for observed use errors, and recommendations for next generation releases or refreshes based on human factors principles.

 

What is Formative Usability Testing?


What is a formative usability test?

Formative usability testing is an evaluation method that helps to “form” the product or website design. Representative users take part in exploratory test sessions where they are asked to perform representative tasks on actual and/or prototype products or websites.

 When should it be used?

Formative usability testing is typically performed iteratively during design and development phases to identify and resolve issues associated with the user interface design that impact efficiency, effectiveness, and satisfaction. Oftentimes usability issues can be identified as part of an initial formative test, design changes can be made to resolve or mitigate those issues, and the product or website can then be retested as part of a second formative test to ensure the usability issues have been addressed appropriately.

 What do you get?

The output from a formative usability test typically includes documentation of the findings for each task and recommendations for resolving or mitigating usability issues identified in the user interface design. Performance and satisfaction metrics can be used to establish success criteria for summative testing. Improvements are typically implemented quickly and further (iterative) testing is done to verify the usability of the improvements and to support continuous forward progress. Additional outputs from formative testing may vary based on the needs of the design and development teams.

What is an Online User Survey?


What is a user survey?

An online user survey is similar to a structured interview with users but completed online and without an interviewer. A list of questions is displayed online and users’ responses are recorded.

 When should it be used?

Online user surveys can be used anytime during a product or website design and development lifecycle. They are a fast and affordable way to collect large amounts of data from a large number of participants. However, it is helpful to have very specific questions that need to be answered and you should be more interested in collecting the who and what such as demographics and quantifiable opinions, preferences, and/or behaviors as opposed to learning the why.

Results can be used to help inform product and website design based on users’ or groups of users’ responses. Online user surveys are used to determine demographic information to better detail the user population. They can also focus on users’ opinions and overall impressions or be used as benchmark studies.  For example, the same online user survey can track opinions over time by being launched every year.

 What do you get?

The output from a user survey typically includes a report that details:

  • The primary goals of the user survey
  • Questions asked to achieve those goals
  • Findings in terms of answers to the survey questions, common themes, and trends
  • Descriptive statistics as well as tabular and graphical displays to help clarify findings
  • Recommendations based on findings

What is a User Interview?


What is a user interview?

A user interview is a semi-structured or structured interview that may be conducted in person or over a phone to collect in-depth information about user needs, goals, experiences, attitudes, and opinions.

 When should it be used?

User interviews are typically performed at the beginning of the design phase. Although rich data from small sample interviews is not generalizable to the larger population, it can be used as a starting point for developing quantitative studies or to explain quantitative results. User interviews can also be used to collect demographic information to better detail the user population. Results can be used to help inform the design based on users’ or groups of users’ opinions and overall impressions.

 What do you get?

The output from a user interview typically includes a report that details:

  • The primary goals of the interview
  • Questions asked to achieve those goals
  • Participant demographics
  • Statistics on the quantitative data from rating, ranking, or other closed-ended questions
  • Common themes or trends from open-ended questions
  • Recommendations based on findings

If desired, user interviews can be recorded and/or transcribed.

What is a Focus Group?


What is a focus group?

A focus group is a guided discussion where a moderator leads an interactive group of representative users through a series of questions and activities focused on specific product or topic.

When should it be used?

Focus groups are typically performed early in the product or website planning process to inform requirements definition and design. Focus groups are great for gaining a better understanding of user opinions, preferences, attitudes, and reactions to a particular topic, concept, or prototype.

What do you get?

The output of a focus group typically includes findings and recommendations that inform requirements, feature offerings, and/or the design of a new or available product or website. Outputs also include insight into who the users are, their characteristics, environments of use, and how the product or website fits (or does not fit) into their lifestyle.

What is a Card Sort?


What is a card sort?

Card sorting is a usability evaluation method used to design and/or evaluate the information architecture for a product or website. Card sorting can also be used to identify logical groupings for toolbars and menus. During a card sort, representative users are provided cards containing items to sort into meaningful groups or categories.

There are different types of card sorting methods:

  • Open Card Sort – Representative users to sort cards into groups that they label.
  • Closed Card Sort – Representative users sort cards into pre-labeled groups.
  • Inverse/Reverse Card Sort or Tree Testing – Representative users are asked to find specific information or topics that have already been sorted into labeled groups.

 When should it be used?

Card sorting is typically performed during the design phase for a product or website when the information architecture is still being developed. However, it can also be used during the development phase or after a product or website has been released to identify if there are potential usability issues associated with how information is grouped or labeled.

What do you get?

The output of a card sort typically includes findings and recommendations associated with groupings and/or labels that can be used to define or validate the information architecture of a product or website and improve discoverability of information.

What is a Heuristic Review?


What is a heuristic review?

A heuristic review is a usability evaluation method where one or more trained usability/human factors experts review a product (hardware and/or software user interfaces) or website and compare it against established human factors criteria and principles for design to identify potential usability issues.

 When should it be used?

Heuristic reviews can be conducted at any time during the design and development phase of a product or website. However, the earlier a heuristic review can be performed (preferably early in the design phase), the more likely it is that identified potential usability issues can be resolved prior to product release.

 What do you get?

The output of a heuristic review typically includes a list of potential usability issues that are categorized and assigned a severity rating based on perceived usability risk and overall impact on the user experience.

Actionable and prioritized recommendations for resolving or mitigating potential usability risks are provided for all issues identified as part of the heuristic review. Visual depictions (e.g., annotated screen captures) are often included to help illustrate findings.

Results of heuristic reviews are presented to product teams in an easily digestible format that can be used internally to track status, add comments, and assign priorities.

Safety-Enhanced Design (Usability) Reports Available…Now What? Finding Reports


I am so distracted today…

First, the build up to the snow and ice event in NC is exhausting. Just let the snow and ice fall; and let the power go out already.

Second, ONC made the Safety-Enhanced Design (Usability) reports public. Thanks @HealthIT_Policy for the link.

Now what?

Well, I know I immediately went to look at some reports. Good luck with that. It seems others must have been having issues as well. Which might have led @Farzad_MD  to repost the original link and to post a link directly to one of the reports.  Thanks for the links.

We are a consulting company and have conducted User Centered Design (UCD) activities and summative tests on several of the certified products. So, I let our clients know the reports are now public. Some of those clients already emailed me back asking how to find the actual reports for their own products and for other products.

If you are having trouble finding UCD and summative test reports on the site… here is a cheat sheet:

 

Here is what I have learned:

  • It seems that products certified through Drummond are most reliable for providing both the UCD report and the Summative Test report.
  • It seems that no products certified through CCHIT have associated reports.
  • Products certified through ICSA Labs have some products with and some products without associated reports.
  • Products certified through InfoGard have some products with and some products without associated reports.

 

Now what…let’s get the discussion started:

  • How can these reports be used in a productive way?
  • How can the data in these reports be used to make EHRs safer and more efficient in support of providing quality care?
  • How can the data in these reports inform future policy?

 

More to come on these topics.

Usability Testing: 5 Strategies to Achieve Meaningful Use Stage 2 Certification


I was recently invited to join Lorraine Chapman and Anneliis Tosiine from Macadmian to be a guest contributor on a podcast about Meaningful Use Stage 2 usability testing. Lorraine and Anneliis are fellow HIMSS Usability Task Force members. The podcast discusses 5 strategies to employ when conducting usability testing for Stage 2 certification. Each of the five strategies stem from User-View’s and Macadamian’s experiences and best practices employed while supporting EHR vendors through the Safety Enhanced Design certification processes.

Usability Testing: 5 Strategies to Achieve Meaningful Use Stage 2 Certification

  1. Planning is Key
    • Obtain prompt access to the EHR that will be tested as part of summative testing. Use this access to familiarize the usability test team with the application.
    • Identify the most efficient click path and all alternative paths that can be used by end users to successfully complete tasks.
    • Work with expert users to identify the optimal completion time for each of the tasks used in the summative test session.
    • Plan for at least 4-6 weeks for a summative test when you have 2 -3 user groups.
  2. Determining Types and Number of Users
    • Start by identifying who your end users (or intended end users) are for specific usability test tasks.
    • Recruit end users who would do the task in the clinical world.
    • Determining number of participants for a summative test is driven by the User Centered Design process your team follows. Human factors best practice is 15 participants per user group are sufficient if you have been doing User Centered Design throughout design and development.
  3. Conducting Remote Usability Testing
    • Usability testing can include remote testing.
    • Remote usability testing (where the facilitator and the participant can be in different locations) can be a great way to reduce cost and expedite the recruiting and data collection process.
    • Keep in mind that if you choose to do remote usability testing, you will need a reliable web conferencing tool so you can view (and record) the sessions in real time.
  4. Importance of Risk Assessments
    • Avoid jumping right into summative testing with end users – do a risk assessment first.
    • Safety enhanced design requires a risk assessment both on the actual tasks/user interfaceassociated with the task you will carry out in your summative test and on the findings in yoursummative test report.
    • A goal of ONC and also for Human Factors / UX Specialists is to ensure the system protectsthe provider and the patient from any errors that might occur due to the user interface. The risk analysis will help identify and mitigate these kinds of problems.
  5. Preparing to Submit to ONC
    • In order to be considered for certification, teams must submit information from summative usability testing. As part of the summative test report, teams are to report the findings of the risk analysis of the use, tested performance, error rates, and satisfaction ratings. Teams are also required to report any critical use errors and specify how the product will rectify these errors. Critical use errors do not need to be reported if the product is redesigned, retested with another summative test, and retesting no longer finds the critical use error is present.
    • An objective, trained human factors researcher / UX specialist should be deeply involved in the preparation of the report for ONC. This is important because summative test reports for ONC will be released publically and you want to be sure that the study is represented accurately and the integrity of the study is maintained when it is shared with the industry.
    • NISTIR 7742 is an important resource to use for reporting.

Listen Now: Macadamian | Usability Testing: 5 Strategies to Achieve Meaningful Use Stage II Certification

You will be required to register with Macadamian to listen to the podcast.

To register, scroll to the bottom of the webpage and click the “Register” link.

The Value Comparing Similar User Needs Across Industries


While every industry has its unique user needs, you can get a lot of value from comparing different industries that on the surface wouldn’t seem to have much in common.

Two independent user-centered design projects were carried out to inform new product development. One project targeted medication inventory management of controlled substances. The other project targeted inventory management of lottery products like scratch tickets and lottery tickets.

Reflecting on the observation and interviews from each project, it became clear the two industries shared user roles, user needs, and workflow. In addition, the two industries shared terminology in how they described their needs, tasks, and workflows.

Medical inventory management

Administrators of pharmacies tasked with inventory management of controlled substances describe that a major problem is loss of product (narcotics) through losing products (e.g., pill slipped into a crevice and cannot be retrieved), damage (e.g., the blister pack is torn), and theft by employees.

Because of the high stakes of loss product (narcotics), pharmacy administrators have a need to carefully track inventory. In fact, tracking must occur at the end of each shift in order to identify any product loss and to begin the reconciliation process as soon as possible.

Throughout a shift, workers must account for each access to the product (narcotics). Employees are required to record missing inventory immediately. In the case of pills fall into crevices and damaged inventory, employees have specific procedures regarding retrieving, storing, and reporting each pill. At the end of each shift, a reconciliation process takes place.

Should a discrepancy in actual versus projected count (based on information recorded) occur, then a reconciliation process begins which includes steps such as identifying each employee who had access to the system throughout the shift, double checking counts and medication orders taken throughout the shift, interviewing each employee who worked during the shift, etc. Sometimes interviews are delayed because an employee is off shift and my not return to work for a day or more.

If discrepancies are not immediately resolved, additional activities take place. Losses are tracked over time and patterns of discrepancies are noted so as to inform future reconciliation processes.

Lottery product inventory management

Turning to stores that sell lottery products, a similar story emerged during the user-centered design process. Store owners tasked with inventory management of lottery products describe that a major problem is loss of product (scratch tickets, lotto tickets, and cash) though loosing products (e.g., a scratch ticket was mistakenly detached from the roll and stuck under the cash drawer), damage (e.g., a scratch ticket was torn), and theft by employees.

Because of the cost of loss product to the store owner, store owners carefully track lottery inventory. In fact, tracking occurs at the end of each shift in order to identify any product loss and to begin the reconciliation process as soon as possible.

Throughout a shift, workers must account for each access to the product (lotto tickets, sale of scratch ticket). Employees are required to record missing inventory immediately. In the case of tickets being removed from the roll by mistake or damaged inventory, employees have specific procedures regarding retrieving, storing and report each ticket. At the end of each shift, a reconciliation process takes place.

Should a discrepancy in actual versus projected count (based on information recorded) occur, then a reconciliation process begins which includes steps such as identifying each employee who had access to the tickets throughout the shift, double checking counts and ticket sales from throughout the shift, interviewing each employee who worked during the shift, etc. Sometimes interviews are delayed because an employee is off shift and my not return to work for a day or more.

If discrepancies are not immediately resolved, additional activities take place. Losses are tracked over time and patterns of discrepancies are noted so as to inform future reconciliation processes.

Comparing industries

As you can see, both industries display similarities.

[what is the value of making this comparison?]