A comparison of hierarchical browsing interfaces : WebTOC

Pilot Results

Four subjects were tested on March 15 at the HCI laboratory in the A.V. Williams building, two per experimental treatment. The treatments tested were: WebTOC with size bars, and no WebTOC. Due to lack of subjects, we were unable to test the treatment involving WebTOC without size bars. However, we feel that this omission is not very significant as the difference between WebTOC with or without size bars is not great. The subjects who participated in the study were all experienced users of computers and the World Wide Web. Only one had visited our experimental site before, though she did not rate her familiarity with the site very highly.

There were several observations that were made over the course of the pilot experiment (for a detailed list, please see appendix). We discovered that there are some technical details that need to be taken care of. For example, we need to customize the Netscape window prior to the experiment so that the current URL and the directory buttons are not shown. This would limit subjects to the functionalities provided by WebTOC and/or the more basic functions of Netscape (Back, Forward, etc.). The followed links need to be reset after each experiment so that one subject will not benefit from the efforts of the previous one.

Further, we formalized our training procedure. We shall start with a quick description of the project, assure subjects that it is the interfaces, not them, that are being tested, review some basic Netscape functionalities, and explain the tools being used for the experiment, namely the timing mechanism and WebTOC. We also discovered that one of the sample tasks we gave subjects was considered hard. We need to review and perhaps modify this task.

The preliminary questionnaire and user satisfaction survey we gave users need some revision, which we shall do prior to the experiment. In particular, the latter document needs to have some reminders of the tasks performed by the users, so they can rate the difficulty level of the same.

The wording of the tasks we assigned the subjects was revised prior to the pilot study. In addition, two of the tasks were changed for the study. We found that the subjects had difficulty with the wording of the following question:

Which of these two photographers have more works included in the collection:
William Henry Jackson, or Carleton E. Watkins?

The complete list of tasks is provided below:

1. Is Moosehead Lake included in the subject index?
2. Which index contains more entries, the author index or the subject index?
3. There is one picture of the Potomac River in this collection. Where exactly was this picture taken?
4. Which of these two photographers have more works included in the collection: William Henry Jackson or Carleton E. Watkins?
5. Between 1872 and 1889, a document entitled "The Extermination of the American Bison" was published. Who was the author of this document?
   

We are considering adding a sixth task that involves searching for images in the web site.

The subjects did not start their search in the author index, as we had anticipated, because they did not associate photographers with authors. Also, we discovered that it is possible to answer this question correctly without all the information required, due to the nature of the site. This question may need revision.

The completion times were extremely high for tasks three and four, though this was encouraged by the lack of a maximum time per task. After reviewing the times, we decided to set the maximum time to be 5 minutes.

We estimate that the experiment will take about 45-50 minutes per group of subjects. The breakdown is as follows:

• Preliminary Questionnaire - 3-5 minutes
• Training - 10 minutes
• Tasks - 30 minutes
• User Satisfaction Survey - 3-5 minutes
  

We feel that we have a good foundation on which to conduct our experiment based on the results of the pilot study.

Preliminary Questionnaire:

• need to state the name of the site
• put spaces between Lycos and Webworm
  

Setup:

• Turn off Location line in Netscape
• Turn off Directory buttons in Netscape
• Between experiments "expire" followed links in Netscape
• Set up WebTOC w/out search and w/out bar changing options
  

Training:

• start with a quick description of the project
• state "We are not testing you, we are testing the interfaces"
• should have detailed "script" to follow
• should use the timer with training task to give subjects familiarity
• review basic Netscape functionality (e.g. Back, Home buttons)
• explain that keyboard (and thus search, etc.) is not allowed
• for Web-TOC, teach the use of the Expand & Contract buttons
• for some reason, the find Children's Event task was hard. Why?
• maybe use another Library of Congress site instead of Enoch-Pratt?
  

Tasks:

• Between tasks, Netscape should be set to top level page
• Subjects should write answer before pressing "Pause", otherwise they might forget it
• In general, the photographer question was difficult b/c people don't think of photographers as "authors"
• With Web-TOC, the photographer question was difficult b/c of loss of context. One subject saw only four names under this heading & skipped it.
• need to somehow prevent/discourage guessing. Also, it is possible to answer question 4 correctly w/out finding all relevant info (one subject thought one photographer only had two works b/c he found him in the subject index
• One Netscape user didn't key off of the dates for the "Bison" task. Found the answer using subject index.
• Need some sort of image finding task
  

User Satisfaction Survey:

• need to include short reminders of each task performed