One from Column A and one from Column B: Simultaneous vs. Sequential Menus

Experiment

Introduction

Our experiment involves two independent variables: menu type and question type. The corresponding dependent variables are speed of task performance, and subjective user satisfaction. However, our main objective is to explore the effects of menu types on performance. In our study, we are comparing simultaneous vs. sequential menus by introducing three types of questions. The classification depends on how much backtracking the user has to do. In other words, the height of the hierarchy is important.

Figure 2.1 is an example of data organization using sequential menu representation. Each screen represents the user's new choice.

  

Figure 2.1: Sequential layout

The simultaneous menu model (fig 2.2) has all the choices presented on the left side of the screen. They remain visible at all times.

  

Figure 2.2: Simultaneous layout

Question types

The questions had three levels of difficulty with respect to the amount of backtracking the user would have to do for sequential menu layout.

Type 1 questions did not make any difference in terms of clicking because no backtracking was necessary for users involved in sequential menu experiment.

How many people were employed in Kent County in service businesses during 1993?

Type 2 questions require a one-step backtracking:

During which year (1995 or 1996) did Caroline County have the largest number of people working in Manufacturing?

In this example, subjects would have to click on the Return to Years link to go back to previous menu.

Type 3 questions require a two-step backtracking:

Which business category employed the larger number of people in Howard County in 1995: manufacturing or wholesale trade?

In this case, the users would have to go back to Years then to Categories in order to choose another industry.

Menu Layouts

Sequential Menus

The sequential menus display choices on a series of three screens: county, category, and year. Each screen has a header that specifies the choices made thus far, in a colon-separated format (County:Category:Year). The industry and year menus include two types of backtracking choices: a ``Return to Counties'' or ``Return to Categories'' link that returns the user to the previous menu, and a ``reset menus'' link that returns the user to the first menu. Once choices have been made in all three categories, the user will be presented with a data screen that displays the number of employees, annual payroll, and number of establishments for the chosen county, industry, and year. This result screen also contains a link for backing-up one step (``Return to Years'') and a ``reset menus'' link.

Simultaneous Menus

The simultaneous menu arrangement displays county, category, and year menus in a series of three frames arranged vertically down the left-hand side of the screen. The right-hand side of the screen is completely occupied by a single frame which contains data.

When the user makes a choice in one of the three menus, that menu is updated in order to indicate the specific choice made. Specifically, the title of the frame is changed to include the name of the choice, and a red diamond is placed next to the choice.

The data window is initially blank, with the exception of a title that reads simply ``data''. As choices in the menus are made, the title is updated update these choices, and a message is displayed, asking the user to make a choice in the other menus. Once choices have been made in all three menus, the data window is updated to display the appropriate facts. At all times, a ``reset menus'' link in the data window can be used to restore all of the menus to their original, unselected state.

Hypothesis

Our null hypothesis states that there is no difference in task completion time between simultaneous and sequential menus, while our alternative hypothesis is that performance with simultaneous menu is faster.

H₀: Mean task performance times for sequential menus and simultaneous menus are equal.

H_A: Mean task performance times for sequential menus is greater than mean task performance times for simultaneous menus.

Since or alternative hypothesis indicates an ordering between the two means, a directional (one-tailed) t-Test has been chosen. The 5% level of significance (alpha = 0.05) will be used.

Pilot study results

Summary

Pilot tests were conducted with four users - two for each treatment (sequential and simultaneous menus). In addition, two of the participants in the project (Natasha and Geoffroy) completed the tasks involved in the sequential menu treatment. All participants were able to finish their tasks in 25 minutes or less.

The pilot tests identified several issues that had to be addressed. These changes can generally be divided into three areas: clarifications of instructions, clarification of task presentation, or modifications to the layout and content of the menu screens.

Modifications

The following modifications have been identified as appropriate. All of these changes were implemented before the actual user testing began.

Instructions

1.

The instructions were modified to encourage participants to answer questions in the order presented.

2.

The instructions were modified to indicate to the users that they should press the Next question link once they have answered a question. They should then read the question and hit the Next question link in order to continue. The links on the menus were updated to match these instructions.

Task Presentation

1.

The relevant fields (county name, industry, statistic type, and year) in the questions were emphasized (bold-face) in order to facilitate comprehension.

2.

The phrasing ``business establishments'' was used in questions involving the number of businesses in a given industry, year, and county. Original phrasings involving only the word ``businesses'' were judged to be confusing.

Menu Layout

1.

The Start and Next hyperlinks were presented in a more distinctive manner.

2.

The Next question link and the ``start'' title on the start screen were placed on the same line.

3.

Text instructing the user to ``click `next question' when you're ready to answer the next question'' was added to the start screen.

Subjects

Twenty two subjects participated in the experiment. Data for two subjects was not used due to abnormally high task completion times. Of the remaining twenty subjects, 15 were male, 5 were female, and all were under 45 years of age. All subjects were graduate or undergraduate students, and all had previous web-browsing experience. Eleven subjects used simultaneous menus while the other nine used sequential menus.

Materials

In order to conduct our experiment, we created our own database with data taken from the Census Bureau Web Site . The data that we needed was hierarchically organized into:

• County (23),
• Industry (9),
• Year (4),
• Data.

with respect to the state of Maryland.

The whole experiment was conducted on the Internet, thus we had to convert all the data into HTML files. We installed an Apache server on two Windows NT Machines. The servers ran locally, thus eliminating false results due to network delays or unexpected interference. We used Netscape as our Web browser with its cache cleared before running the experiment. The server was configured to automatically log the times for each click the user did. A parser was written to retrieve the necessary information from these logs.

The time recorded is one full question: from the first click to the last. We asked the subjects to write the answer on their answer before the last click.

We wanted to extract the number of clicks per question but the frame design in the simultaneous layout discouraged us. We could still detect higher number of clicks.

In addition, we thought it would be interesting to see the effects on speed of performance over time. So, all of us ran the experiment on ourselves (three times per each version). For each of the 15 questions, the best time between all these runs was selected. Then we aggregated the questions by question type.

Procedures and Problems

The subjects were divided into two experimental groups. They were randomly assigned to either work with simultaneous or sequential menus. The description of the procedures was different for each set of subjects (simultaneous, sequential), but the questions were the same.

The experiment session began with subjects signing the User Agreement Form. After that, they were asked to fill in background information. Then, description of the tasks and practice questions were presented to them. The answers had to be recorded on a piece of paper. No help was provided during the real experimentation. Upon subjects' completion, they had to fill out the Subjective Satisfaction Questionnaire.

During our review of the logs, we realized that data for 2 subjects who used sequential model had to be thrown out. Their time values were significantly different from the rest of the participants. Unfortunately, it is not possible to determine whether the error came from produced by server malfunction or subject's confusion.

A predictive model

Our expectation is that simultaneous menus will lead to faster task performance than simultaneous menus and that performance advantages of simultaneous windows will be greater for tasks involving more backtracking through the menu structure. A simple model based on the number of clicks required for each tasks type will help provide a greater understanding of the predicted performance differences.

• For both menu configurations, Type 1 tasks require one selection at each of the three menus, for a total of three clicks.
• Type 2 tasks require one selection at each of the three levels, plus appropriate clicks to get the comparison point. For simultaneous menus, this involves one additional click on the third menu, for a total of 4 clicks. For sequential menus, one click of the Back button is required, along with one additional click on the third menu, for a total of 5.
• Type 3 tasks also require 4 clicks for simultaneous menus, as these questions still only alter one of the three menu choices. However, sequential menus require 7 clicks: 5 as required for type 2, plus one more Back click and a new menu selection at the second level.

These results are summarized in table 2.1. The ``Items Varied'' for any given task is the total number of menu choices that changes. It should be noted that for type 3, only one item is varied, even though two backtracking steps are required in the sequential case. Examination of the type 3 questions reveals the reason for this: these questions require a different choice on the second menu, while requiring the same choice for both visits to the third menu.

 

 

Task # Varied Simult. Clicks # Backtracks Sequent. Clicks

1 3 3

2 1 4 1 5 (=3+1+1)

3 1 4 2 7 (=3+2+2)

For simultaneous menus, the number of items varied in the task is clearly the most important factor. If we assume a menu structure with j menus and n_i choices at level i, then a task involving k varied items will require

$$\sum_{i=1}^{j} E(n_{i}) + \sum_{i=j-k}^{j} E(n_{i})$$ (1)

where E(n_i) is the expected amount of time that will be spent on making a selection from a menu containing n_i choices. In other words, the total time required is the time needed to make the initial j choices and then k additional choices.

For sequential menus, the number of backtracking steps is the factor that differentiates between the tasks. Each backtracking step requires one selection of the Back button, and one additional menu selection. So, given the same number of menus and number of items on each menu, a task with k backtracking steps will require

$$\sum_{i=1}^{j} E(n_{i}) + k \times r + \sum_{i=j-k}^{j} E(n_{i})$$ (2)

where r is the amount of time required to hit the Back button.

For any given combination of a set of tasks, a menu organization, and a question involving comparison among different values, the number of items varied will always be less than or equal to the number of backtracking steps. To see why this is so, consider the general case: if i menu items are varied, at least i backtracking steps will be required to see the i menus necessary to make the varied choices. Of course, the sequential menus may suffer from cases where i items are varied but the number of backtracking steps require is greater than i.

Sequential menu performance also suffers from inclusion r, the time required to hit the Back key. While this may be seen as putting sequential menus at a disadvantage, it seems to be an inherent structural property of sequential menus.

This analysis assumes that the selection times E(n_i) are comparable for the two types of menus. Past research has shown that menu selection times can be roughly logarithmic or linear [Norman, 1991]. Although our experiment does not attempt to build a complete performance model, it seems reasonable to assume that the type of the selection times (linear or logarithmic) function will be the same for both arrangements.