Experiment

2.1 Introduction and Hypothesis

Many large websites create hierarchical tree structures, breaking information down into categories and subcategories. These sites often provide a navigation bar that allows users to backtrack to any point in the path of visited pages. In order to explore this issue, we set up an experiment to find out the most efficient navigation bar structure. Our experiment compares the standard navigation bar to the selection list navigation bar. We created a test website for each navigating structure, and subjects were asked to find information within the website. For each type of navigation bar, we measured and compared completion times and number of page loads for subjects to complete a set of multiple choice questions.

We hypothesize that the subjects will complete search tasks in less time using the selection list navigation bars than using the control or standard navigation bars. We believe that the subjects will also load fewer web pages using the selection list navigation bars. As a result of the increased efficiency, we think that the subjects will prefer the selection list navigation bars over the basic and standard navigation bars.

The experiment will have one independent variable: navigation bar type. The three treatments are the control (no navigation bar just a link back to the root page), the standard navigation bar, and selection list navigation bar.

The experiment will have three dependent variables: time to complete tasks, number of web page loads to complete tasks, and users' subjective preference of navigation bar treatments.

We conducted a pilot study of our experiment on five subjects. They were college students who were comfortable using a computer and had no difficulty using a mouse to perform the tasks. After running the pilot study, we found several improvements that needed to be made:

• The multiple-choice answers needed to be unselected at the beginning of each question.
• The first two questions differed by only two words, and the answer choices were identical. Some of the subjects did not notice that the question had changed, so they clicked the "Next" button a second time and accidentally skipped the second question. We corrected this problem.
• Some of the questions told the subject which part of the tree to go to while others did not. All of the questions needed to be consistent.
• Originally, we asked five random questions for each treatment in the pilot. We changed the questions so that in each treatment the answers were in the same relative locations in the three tree hierarchies.
• Subjects seemed to answer the last two questions of the experimental treatment much faster than the first three. Thus, we used ten questions per treatment in the actual experiment in case there was a short learning curve.

Based on our observation of the pilot study, we noticed some of the subjects began to do other things during the experiment. In our experiment, we made sure subjects remained on task by reminding them at the beginning that they were being timed.

For our experiment, we found 18 subjects who were capable Internet users (familiar with the standard navigation bar), but not computer science experts. We required that the subjects not be computer science undergraduates or graduates. We ran a 3x1 within-subjects experiment. With 3 treatments of the independent variable, we tested subjects in each of the 6 possible orderings of treatments to counterbalance the experiment. Therefore, we tested 18 subjects, giving us 3 subjects for each treatment ordering.

Our experiment was performed on typical PCs familiar to our subjects. The Visual Basic program that we created simulated a web browser and at the same time displayed the multiple choice questions at the bottom of the screen.

The benefit of this interface was that we limited our subject's navigation possibilities to the navigation bar and the links themselves. There was no available back button, forward button, or other methods to navigate the structure. In addition, a Visual Basic program facilitated easy storing and retrieval of the data. Behind the scenes, the program recorded the path the subjects took, the answers they provided, and the elapsed time. Before beginning, subjects were provided short instructions and a sample task to perform. Following completion of the multiple-choice questions, subjects answered a short questionnaire. They were also provided with a debriefing paper to help explain the purpose of the experiment.

Each treatment trial asked ten different multiple choice questions. Subjects had to navigate three different web hierarchies to answer the questions. However, unknown to the subjects, the answers were at the same relative locations in the three different structures. The Visual Basic program and necessary assisting files can be downloaded in this zip file.

Here are the explicit instructions for conducting the experiment:

1. In Windows, click on Start, Control Panel, Internet Options, and then Clear History so all of the links in the experiment will return to their original color.
2. Close all programs, including virus checkers and instant messengers.
3. Start the quiz1.exe program.
4. Enter the appropriate subject ID number. Either use 1-6, 7-12, or 13-18. Use each of your ID numbers exactly once (doing them in order would make sense). The ID number is used to select the order of the treatments, so this will make sure we test each order of treatments exactly 3 times.
5. Write the subject's ID number on the survey form that you will give them later.
6. Pull the Windows task bar at the bottom of the screen down so the experiment program takes up the entire screen.
7. Turn off the radio, TV, phone, and any other possible distractions.
8. Seat the subject in front of the computer.
9. Instruct the subject to read and sign the consent form. Collect it when they are finished.
10. Instruct the subject to read and follow the instructions on the screen. If they have any questions, they can ask you at any time.
11. Let the subject complete the experiment. DO NOT look over their shoulder as they work.
12. When the subject has completed the experiment, give them the survey form with their subject ID number on it and instruct them to fill it out. Collect it when they are finished.
13. Give the subject a debriefing paper and thank them for participating.
14. Close the quiz1.exe program by clicking the X in the top right corner.
15. The data for that subject is now stored in a file called subjectn.exe where n is their subject ID number. This file is in the output directory.

We had two problems. One particular subject did not realize that the experiment was timed. As a result, the subject performed the experiment while at the same time instant messaging friends. This subject's elapsed time was more than 10 times other subject's elapsed time. We decided to test an additional subject and replace this tainted data. Our second problem was that a few answers were not located in an intuitive location. This caused difficulty finding the answer, independent of the navigation bar treatment. As a result, we performed additional statistical analysis.