Results

Four sets of t-statistics were calculated to analyze the data. In the first analysis, the data from all ten questions of each treatment was compared to the data from all ten questions of each of the other treatments. It was possible that the subjects' use of the navigation bars improved with practice in just the short span of the experiment, so the second analysis compared data from the first five questions of each treatment while the third analysis compared data from the last five questions of each treatment. Each of these first three analyses compared the time to complete the questions, the number of page loads to complete the questions, and the number of questions answered correctly. Finally, the fourth analysis compared the responses provided on the subjective surveys. The raw data collected during the experiment can be found in Appendix B.1, B.2 and B.3, while the raw data collected from the subjective surveys can be found in Appendix B.4.

In the analysis of the number of seconds that subjects took to complete all ten questions, subjects spent the least time answering questions in the control treatment (M=175.55, SD=74.70). This was statistically significantly lower than the mean number of seconds to complete the questions using the standard treatment (M=290.08, SD=98.82), t=7.05, p=0.05, or the mean number of seconds to complete the selection list treatment (M=269.66, SD=92.00), t=5.27, p=0.05. No significant time difference was found between the standard treatment and the selection list treatment, t=1.74, p=0.05. These results are shown in Figure 1.

An analysis of the first five questions revealed that for this subset, task completion times were again significantly lower for the control treatment (M=88.23, SD=43.85) than the standard treatment (M=121.53, SD=50.12), t=3.35, p=0.05, or the selection list treatment (M=157.80, SD=45.55), t=5.29, p=0.05. In these questions, however, the selection list treatment did take a significantly longer time to complete than did the standard treatment, t=3.27, p=0.05. These results are shown in Figure 2.

The results of the last five questions of each treatment were slightly different. Once again, they reveal that the simple treatment took significantly less time (M=87.31, SD=36.58) than the standard treatment (M=168.55, SD=64.46), t=7.27, p=0.05, or the selection list treatment (M=111.86, SD=56.26), t=2.54, p=0.05. In contrast to the results from the first five questions, subjects spent significantly more time answering the last five questions of the standard treatment than they did the selection list treatment, t=4.98, p=0.05. The time results of the last five questions are shown in Figure 3.

It appears that the time differences between the first five questions and the last five questions for the standard and selection list treatments offset each other, leading to no significant time difference between those treatments for all ten questions. The raw time data can be found in Appendix B.1.

The mean number of page loads to complete all ten questions using the standard treatment (M=71.27, SD=16.16) was significantly higher than the mean number of page loads to complete the questions using the control treatment (M=47.61, SD=12.58), t=5.08, p=0.05, or the selection list treatment (M=42.39, SD=10.71), t=8.27, p=0.05. There was no significant difference in the mean number of page loads between the control treatment and the selection list treatment, t=1.62, p=0.05. The page load results of all ten questions are shown in Figure 4.

In the analysis of just the first five questions, no significant differences were found in the number of page loads between any of the treatments. The mean number of page loads (with standard deviations in parentheses) for the control, standard, and selection list treatments are 22.83 (5.75), 25.77 (5.32), and 24.61 (7.01), respectively. These results are shown in Figure 5.

The analysis of the number of page loads from the last five questions showed significant differences between all of the treatments. Subjects loaded significantly fewer pages in the selection list treatment (M=17.77, SD=4.58) than in the control treatment (M=24.77, SD=8.91), t=3.84, p=0.05, or in the standard treatment (M=45.50, 13.98), t=8.81, p=0.05. Subjects also loaded significantly more pages in the standard treatment than in the control treatment, t=5.40, p=0.05. These results are shown in Figure 6. The raw data for the number of page loads can be found in Appendix B.2.

Finally, no statistically significant differences were found in the number of correct answers between any of the treatments for all ten questions or for either of the subsets of questions analyzed. Subjects gave incorrect responses very infrequently. Although the number of correct answers was not a primary interest in the study, the results indicate that the questions were simple enough that subjects spent most of their time looking for answers rather trying to understand the questions. The means and standard deviations of the number of correct answers are listed in Table 1, and the results are shown in Figure 7. The raw data for the number of correct answers can be found in Appendix B.3.

The survey results did not yield many significant differences. Below is a list that summarizes the findings from each of the seven subjective response questions on the survey. The means and standard deviations of the survey results are in Table 2, and the results are shown graphically in Figure 8. The raw data for the survey responses can be found in Appendix B.4.

1. Confusion - Subjects felt significantly more confused by the selection list treatment than by the control treatment.
2. Frustration - Subjects were significantly more frustrated by a lack of links in the control treatment than in the standard treatment.
3. Helpfulness - No significant difference was found in subjective ratings of helpfulness for each of the treatments.
4. Understanding - Subjects reported significantly less understanding of the web site structure with the control treatment than with the standard treatment.
5. Comfort - Subjects felt significantly less comfortable using the selection list treatment than using the control treatment.
6. Practice - Subjects felt that they would like the selection list treatment more with more practice significantly more than the control treatment.
7. Future use - No significant difference was found in the subjects' desire to use the different styles of navigation bars in the future.