blog


Happy World Usability Day 2014


Today, Thursday, November 13th is World Usability Day.  This year’s team is Engagement: www.worldusabilityday.org. Be sure and check out events online and in your neighborhood.  If you are online check out World Usability Day Challenges from the HIMSS Health IT Usability Committee! If you are in my neighborhood, check out the TriUXPA Event at SAS Campus in Cary, NC! What can you do today to improve the engagement we have with technology products and services?

What is Summative or Validation Usability Testing?


What is a summative test?

Summative testing is a comprehensive usability evaluation of a product or website in the context of how it is intended to be used in an actual use environment. Representative users take part in summative test sessions where they are asked to perform representative tasks under realistic conditions on final or near-final products or websites.

When should it be used?

Summative testing should be performed near the end of the product development phase on a final or near-final product or website. The goals of summative testing are to establish benchmarks for efficiency, effectiveness, and satisfaction; and to determine if the product or website has successfully met usability goals / success criteria that were established during formative testing.

What do you get?

The output from a summative test typically includes documentation of the findings for each task including performance summaries, a determination of whether or not usability goals / success criteria have been met for the product or website, a use error analysis discussing the results organized around a risk analysis of use, mitigation strategies for observed use errors, and recommendations for next generation releases or refreshes based on human factors principles.

 

What is Formative Usability Testing?


What is a formative usability test?

Formative usability testing is an evaluation method that helps to “form” the product or website design. Representative users take part in exploratory test sessions where they are asked to perform representative tasks on actual and/or prototype products or websites.

 When should it be used?

Formative usability testing is typically performed iteratively during design and development phases to identify and resolve issues associated with the user interface design that impact efficiency, effectiveness, and satisfaction. Oftentimes usability issues can be identified as part of an initial formative test, design changes can be made to resolve or mitigate those issues, and the product or website can then be retested as part of a second formative test to ensure the usability issues have been addressed appropriately.

 What do you get?

The output from a formative usability test typically includes documentation of the findings for each task and recommendations for resolving or mitigating usability issues identified in the user interface design. Performance and satisfaction metrics can be used to establish success criteria for summative testing. Improvements are typically implemented quickly and further (iterative) testing is done to verify the usability of the improvements and to support continuous forward progress. Additional outputs from formative testing may vary based on the needs of the design and development teams.

What is an Online User Survey?


What is a user survey?

An online user survey is similar to a structured interview with users but completed online and without an interviewer. A list of questions is displayed online and users’ responses are recorded.

 When should it be used?

Online user surveys can be used anytime during a product or website design and development lifecycle. They are a fast and affordable way to collect large amounts of data from a large number of participants. However, it is helpful to have very specific questions that need to be answered and you should be more interested in collecting the who and what such as demographics and quantifiable opinions, preferences, and/or behaviors as opposed to learning the why.

Results can be used to help inform product and website design based on users’ or groups of users’ responses. Online user surveys are used to determine demographic information to better detail the user population. They can also focus on users’ opinions and overall impressions or be used as benchmark studies.  For example, the same online user survey can track opinions over time by being launched every year.

 What do you get?

The output from a user survey typically includes a report that details:

  • The primary goals of the user survey
  • Questions asked to achieve those goals
  • Findings in terms of answers to the survey questions, common themes, and trends
  • Descriptive statistics as well as tabular and graphical displays to help clarify findings
  • Recommendations based on findings

What is a User Interview?


What is a user interview?

A user interview is a semi-structured or structured interview that may be conducted in person or over a phone to collect in-depth information about user needs, goals, experiences, attitudes, and opinions.

 When should it be used?

User interviews are typically performed at the beginning of the design phase. Although rich data from small sample interviews is not generalizable to the larger population, it can be used as a starting point for developing quantitative studies or to explain quantitative results. User interviews can also be used to collect demographic information to better detail the user population. Results can be used to help inform the design based on users’ or groups of users’ opinions and overall impressions.

 What do you get?

The output from a user interview typically includes a report that details:

  • The primary goals of the interview
  • Questions asked to achieve those goals
  • Participant demographics
  • Statistics on the quantitative data from rating, ranking, or other closed-ended questions
  • Common themes or trends from open-ended questions
  • Recommendations based on findings

If desired, user interviews can be recorded and/or transcribed.

What is a Focus Group?


What is a focus group?

A focus group is a guided discussion where a moderator leads an interactive group of representative users through a series of questions and activities focused on specific product or topic.

When should it be used?

Focus groups are typically performed early in the product or website planning process to inform requirements definition and design. Focus groups are great for gaining a better understanding of user opinions, preferences, attitudes, and reactions to a particular topic, concept, or prototype.

What do you get?

The output of a focus group typically includes findings and recommendations that inform requirements, feature offerings, and/or the design of a new or available product or website. Outputs also include insight into who the users are, their characteristics, environments of use, and how the product or website fits (or does not fit) into their lifestyle.

What is a Usability Test?


There are two types of usability tests;  Formative and Summative.

What is a formative usability test?

Formative usability testing is an evaluation method that helps to “form” the product or website design. Representative users take part in exploratory test sessions where they are asked to perform representative tasks on actual and/or prototype products or websites.

When should a formative usability test be used?

Formative usability testing is typically performed iteratively during design and development phases to identify and resolve issues associated with the user interface design that impact efficiency, effectiveness, and satisfaction. Oftentimes usability issues can be identified as part of an initial formative test, design changes can be made to resolve or mitigate those issues, and the product or website can then be retested as part of a second formative test to ensure the usability issues have been addressed appropriately.

 What do you get from a formative usability test?

The output from a formative usability test typically includes documentation of the findings for each task and recommendations for resolving or mitigating usability issues identified in the user interface design. Performance and satisfaction metrics can be used to establish success criteria for summative testing. Improvements are typically implemented quickly and further (iterative) testing is done to verify the usability of the improvements and to support continuous forward progress. Additional outputs from formative testing may vary based on the needs of the design and development teams.

What is a summative test?

Summative testing is a comprehensive usability evaluation of a product or website in the context of how it is intended to be used in an actual use environment. Representative users take part in summative test sessions where they are asked to perform representative tasks under realistic conditions on final or near-final products or websites.

 When should a summative usability test be used?

Summative testing should be performed near the end of the product development phase on a final or near-final product or website. The goals of summative testing are to establish benchmarks for efficiency, effectiveness, and satisfaction; and to determine if the product or website has successfully met usability goals / success criteria that were established during formative testing.

 What do you get from a summative usability?

The output from a summative test typically includes documentation of the findings for each task including performance summaries, a determination of whether or not usability goals / success criteria have been met for the product or website, a use error analysis discussing the results organized around a risk analysis of use, mitigation strategies for observed use errors, and recommendations for next generation releases or refreshes based on human factors principles.

What is a Card Sort?


What is a card sort?

Card sorting is a usability evaluation method used to design and/or evaluate the information architecture for a product or website. Card sorting can also be used to identify logical groupings for toolbars and menus. During a card sort, representative users are provided cards containing items to sort into meaningful groups or categories.

There are different types of card sorting methods:

  • Open Card Sort – Representative users to sort cards into groups that they label.
  • Closed Card Sort – Representative users sort cards into pre-labeled groups.
  • Inverse/Reverse Card Sort or Tree Testing – Representative users are asked to find specific information or topics that have already been sorted into labeled groups.

 When should it be used?

Card sorting is typically performed during the design phase for a product or website when the information architecture is still being developed. However, it can also be used during the development phase or after a product or website has been released to identify if there are potential usability issues associated with how information is grouped or labeled.

What do you get?

The output of a card sort typically includes findings and recommendations associated with groupings and/or labels that can be used to define or validate the information architecture of a product or website and improve discoverability of information.

What is a Heuristic Review?


What is a heuristic review?

A heuristic review is a usability evaluation method where one or more trained usability/human factors experts review a product (hardware and/or software user interfaces) or website and compare it against established human factors criteria and principles for design to identify potential usability issues.

 When should it be used?

Heuristic reviews can be conducted at any time during the design and development phase of a product or website. However, the earlier a heuristic review can be performed (preferably early in the design phase), the more likely it is that identified potential usability issues can be resolved prior to product release.

 What do you get?

The output of a heuristic review typically includes a list of potential usability issues that are categorized and assigned a severity rating based on perceived usability risk and overall impact on the user experience.

Actionable and prioritized recommendations for resolving or mitigating potential usability risks are provided for all issues identified as part of the heuristic review. Visual depictions (e.g., annotated screen captures) are often included to help illustrate findings.

Results of heuristic reviews are presented to product teams in an easily digestible format that can be used internally to track status, add comments, and assign priorities.

Safety-Enhanced Design (Usability) Reports Available…Now What? Finding Reports


I am so distracted today…

First, the build up to the snow and ice event in NC is exhausting. Just let the snow and ice fall; and let the power go out already.

Second, ONC made the Safety-Enhanced Design (Usability) reports public. Thanks @HealthIT_Policy for the link.

Now what?

Well, I know I immediately went to look at some reports. Good luck with that. It seems others must have been having issues as well. Which might have led @Farzad_MD  to repost the original link and to post a link directly to one of the reports.  Thanks for the links.

We are a consulting company and have conducted User Centered Design (UCD) activities and summative tests on several of the certified products. So, I let our clients know the reports are now public. Some of those clients already emailed me back asking how to find the actual reports for their own products and for other products.

If you are having trouble finding UCD and summative test reports on the site… here is a cheat sheet:

 

Here is what I have learned:

  • It seems that products certified through Drummond are most reliable for providing both the UCD report and the Summative Test report.
  • It seems that no products certified through CCHIT have associated reports.
  • Products certified through ICSA Labs have some products with and some products without associated reports.
  • Products certified through InfoGard have some products with and some products without associated reports.

 

Now what…let’s get the discussion started:

  • How can these reports be used in a productive way?
  • How can the data in these reports be used to make EHRs safer and more efficient in support of providing quality care?
  • How can the data in these reports inform future policy?

 

More to come on these topics.