Tutorial

Using Novice–Expert+ to uncover usability issues with a website

By Karl F. MacDorman on June 1, 2022

Walkthrough

Consider the task of identifying a vacation rental in Indianapolis on Craigslist.

Record the user's actions

  1. Start from https://indianapolis.craigslist.org/.
  2. Press Record in the extension's pop-up.
  3. Click vacation rentals.
  4. Get focus on miles from.
  5. Type 15.
  6. Get focus on location.
  7. Type 46202.
  8. Get focus on Bedrooms.
  9. Type 1.
  10. Press return.
  11. Click Beautifully Updated 1-Bedroom 1-Bath Condo.
  12. Read the listing.
  13. Click reply.
  14. Click email.
  15. Click the email address.
  16. Press Pause in the pop-up.

System response and mentally prepare

Add system response before step 4 and 11. Add mentally prepare before step 3, 4, 8, and 13.

Add mentally prepare before step 11.

Save and open in a spreadsheet

Save the log by clicking on Save as CSV in the pop-up.

Open the log in a spreadsheet.

Calculating the NE ratio

A table similar to the one below will appear in the spreadsheet. It will also include some extra columns, which may be hidden or deleted.

The novice–expert ratio is calculated for each step in a task. A step may include more than one KLM GOMS operator. There are several reasons for this:

  1. Novice–Expert+ can only record user-initiated events. For example, if the user homes hands on the trackpad, mentally prepares, points to a widget, and clicks it, only the click is registered. So, the extension doesn't know when the transitions occurred between homing, thinking, pointing, and clicking and thus cannot calculate the execution time of each of these operators.
  2. It is not useful to decompose a sequence of operators resulting from a single chunk of information. A word, for example, forms a cognitive unit, which results in a sequence of keystrokes when typed. Thus, the sequence should be treated as a unit.
  3. Excessive granularity can make it difficult to explain the results to others.

The first novice–expert ratio is the time it took the novice to click vacation rentals divided by the expert time. We can't separate mentally prepare, point, and click time for the novice, but we know the time taken was 6392. The estimated expert time for home hands, mentally prepare, and point and click is 400 + 1200 + 1292, so the NE ratio is

\[ \frac{6231}{400+1200+1292}=\frac{6231}{2892}=2.21 \]

The next click is to get focus on miles from. The novice is not responsible for time spent waiting for a system response, so this time is deducted from the time taken. Thus, the NE ratio is

\[ \frac{-70 + 3867}{1200 + 1226} = 1.57 \]

The next table entry is home, which indicates homing the hands on a device. The extension has detected a transition from using the pointing device to typing on the keyboard. A row is added for the 400 ms expert estimate.

We don't know how long the user took to home the hands, but we know the time taken to home the hands and type 15 is 2995 + 600 ms. The expert estimate is 400 for homing the hands plus 2 times 200 for hitting two keys. Thus, the NE ratio is

\[ \frac{2995 + 600}{400 + 200 + 200} = 4.49 \]

eventType  nodeText  width  height  centerX  centerY  timeTaken  expertTime  NEratio
home 400
think 1200
click vacation rentals 224.8 18.45 912.29 281.95 6392 1292 2.21
response -70 70
think 1200
click 71.3 23.5 43.15 218.52 3867 1226 1.57
home 400
keystroke 1 2995 200
keystroke 5 600 200 4.49
home 400
click 71.3 23.5 119.45 218.52 2154 958 1.59
home 400
keystroke 4 2525 200
keystroke 6 373 200
keystroke 2 636 200
keystroke 0 374 200
keystroke 2 314 200 3.02
home 400
think 1200
click 78.1 23.5 46.55 343.92 2861 1026 1.09
home 400
keystroke 1 1208 200
keystroke Enter 991 200 2.75
response -70 70
read 20640
home 400
think 1200
click reply 62.27 32 388.64 105.13 24682 738 1.07
click email 200 34.5 458.5 139.38 2883 794 3.63
click @hous.craigslist.org 200 18.5 466.5 203.13 1783 965 1.85

In this manner, we complete the remaining NE ratios.

A row is missing for the estimated expert reading time for 86 words at 250 words per minute, converted to milliseconds:

\[ 86 \times \frac{60 \times 1000}{250} = 20640 \text{ ms} \]

The NE ratio for reading, homing the hands, thinking, and clicking Reply is

\[ \frac{-70 + 24682}{20640+400+1200+738} = 1.07 \]

The average NE ratio is

\[ \frac{2.21 + 1.57 + 4.49 + 1.59 + 3.02 + 1.09 + 2.75 + 1.07 + 3.63 + 1.85}{10}=2.33 \]

Conclusion

The website presented several usability issues. After visiting Craigslist for Indianapolis and selecting vacation rentals, rentals appeared for Florida and other places. The 4.49 NE ratio before narrowing the search reflects disorientation with the unexpected results.

Another surprise was the need to click three times to reply to an ad with the third click opening an external email program. Click email had an NE Ratio of 3.63.