Walkthrough
Consider the task of identifying a vacation rental in Indianapolis on Craigslist.
Record the user's actions
- Start from https://indianapolis.craigslist.org/.
- Press Record in the extension's pop-up.
- Click vacation rentals.
- Get focus on miles from.
- Type 15.
- Get focus on location.
- Type 46202.
- Get focus on Bedrooms.
- Type 1.
- Press return.
- Click Beautifully Updated 1-Bedroom 1-Bath Condo.
- Read the listing.
- Click reply.
- Click email.
- Click the email address.
- Press Pause in the pop-up.
System response and mentally prepare
Add system response before step 4 and 11. Add mentally prepare before step 3, 4, 8, and 13.
Add mentally prepare before step 11.
Save and open in a spreadsheet
Save the log by clicking on Save as CSV in the pop-up.
Open the log in a spreadsheet.
Calculating the NE ratio
A table similar to the one below will appear in the spreadsheet. It will also include some extra columns, which may be hidden or deleted.
The novice–expert ratio is calculated for each step in a task. A step may include more than one KLM GOMS operator. There are several reasons for this:
- Novice–Expert+ can only record user-initiated events. For example, if the user homes hands on the trackpad, mentally prepares, points to a widget, and clicks it, only the click is registered. So, the extension doesn't know when the transitions occurred between homing, thinking, pointing, and clicking and thus cannot calculate the execution time of each of these operators.
- It is not useful to decompose a sequence of operators resulting from a single chunk of information. A word, for example, forms a cognitive unit, which results in a sequence of keystrokes when typed. Thus, the sequence should be treated as a unit.
- Excessive granularity can make it difficult to explain the results to others.
The first novice–expert ratio is the time it took the novice to click vacation rentals divided by the expert time. We can't separate mentally prepare, point, and click time for the novice, but we know the time taken was 6392. The estimated expert time for home hands, mentally prepare, and point and click is 400 + 1200 + 1292, so the NE ratio is
\[ \frac{6231}{400+1200+1292}=\frac{6231}{2892}=2.21 \]
The next click is to get focus on miles from. The novice is not responsible for time spent waiting for a system response, so this time is deducted from the time taken. Thus, the NE ratio is
\[ \frac{-70 + 3867}{1200 + 1226} = 1.57 \]
The next table entry is home, which indicates homing the hands on a device. The extension has detected a transition from using the pointing device to typing on the keyboard. A row is added for the 400 ms expert estimate.
We don't know how long the user took to home the hands, but we know the time taken to home the hands and type 15 is 2995 + 600 ms. The expert estimate is 400 for homing the hands plus 2 times 200 for hitting two keys. Thus, the NE ratio is
\[ \frac{2995 + 600}{400 + 200 + 200} = 4.49 \]
eventType | nodeText | width | height | centerX | centerY | timeTaken | expertTime | NEratio |
home | 400 | |||||||
think | 1200 | |||||||
click | vacation rentals | 224.8 | 18.45 | 912.29 | 281.95 | 6392 | 1292 | 2.21 |
response | -70 | 70 | ||||||
think | 1200 | |||||||
click | 71.3 | 23.5 | 43.15 | 218.52 | 3867 | 1226 | 1.57 | |
home | 400 | |||||||
keystroke | 1 | 2995 | 200 | |||||
keystroke | 5 | 600 | 200 | 4.49 | ||||
home | 400 | |||||||
click | 71.3 | 23.5 | 119.45 | 218.52 | 2154 | 958 | 1.59 | |
home | 400 | |||||||
keystroke | 4 | 2525 | 200 | |||||
keystroke | 6 | 373 | 200 | |||||
keystroke | 2 | 636 | 200 | |||||
keystroke | 0 | 374 | 200 | |||||
keystroke | 2 | 314 | 200 | 3.02 | ||||
home | 400 | |||||||
think | 1200 | |||||||
click | 78.1 | 23.5 | 46.55 | 343.92 | 2861 | 1026 | 1.09 | |
home | 400 | |||||||
keystroke | 1 | 1208 | 200 | |||||
keystroke | Enter | 991 | 200 | 2.75 | ||||
response | -70 | 70 | ||||||
read | 20640 | |||||||
home | 400 | |||||||
think | 1200 | |||||||
click | reply | 62.27 | 32 | 388.64 | 105.13 | 24682 | 738 | 1.07 |
click | 200 | 34.5 | 458.5 | 139.38 | 2883 | 794 | 3.63 | |
click | @hous.craigslist.org | 200 | 18.5 | 466.5 | 203.13 | 1783 | 965 | 1.85 |
In this manner, we complete the remaining NE ratios.
A row is missing for the estimated expert reading time for 86 words at 250 words per minute, converted to milliseconds:
\[ 86 \times \frac{60 \times 1000}{250} = 20640 \text{ ms} \]
The NE ratio for reading, homing the hands, thinking, and clicking Reply is
\[ \frac{-70 + 24682}{20640+400+1200+738} = 1.07 \]
The average NE ratio is
\[ \frac{2.21 + 1.57 + 4.49 + 1.59 + 3.02 + 1.09 + 2.75 + 1.07 + 3.63 + 1.85}{10}=2.33 \]
Conclusion
The website presented several usability issues. After visiting Craigslist for Indianapolis and selecting vacation rentals, rentals appeared for Florida and other places. The 4.49 NE ratio before narrowing the search reflects disorientation with the unexpected results.
Another surprise was the need to click three times to reply to an ad with the third click opening an external email program. Click email had an NE Ratio of 3.63.