Hierarchies on the World Wide Web

Panayiotis Zaphiris

Systems Engineering

Ben Shneiderman

Computer Science Department

Kent L. Norman

Department of Psychology

University of Maryland
College Park, MD 20742

March, 1999

ABSTRACT

An experiment is reported that compared expandable indexes providing full menu context with sequential menus providing only context. Menu depth was varied using hierarchies of 2, 3, and 4 levels deep in an asymmetric structure of 457 root level items. Menus were presented on the World Wide Web within a browser. Participants searched for specific targets. Results suggest that reducing the depth of hierarchies improves performance in terms of speed and search efficiency. Surprisingly, expandable indexes resulted in poorer performance with deeper hierarchies than did sequential menus.

Keywords: Information retrieval, menu selection, depth versus breath in information design, web page design, hypertext, searching, expandable indexes.

Information retrieval from the World Wide Web (WWW) is becoming a daily activity in both work and leisure environments. In order to make information retrieval more efficient, it is necessary that indexes, menus, and links be carefully designed.

The goal of this study was to investigate the use of expanding hierarchical indexes. One advantage of using such indexes for menu selection is that they preserve the full context of the choice within the hierarchy. While the user browses through the hierarchical structure, the tree is fully displayed. Thus at any point, the user has access to the whole set of major and same level categories. Sequential menus, on the other hand, do not display the full hierarchical context as they drop down to deeper levels in hierarchy. Only elements in the selected category are displayed as options for browsing. This is of particular importance on the WWW when the number of root level alternatives and the depth of the hierarchy can be large.

Depth versus breadth in hierarchical menu structures has been the topic of much research. The trade-off between menu depth and breadth is considered by some researchers as the most important aspects that must be considered in the design of hierarchical menu systems (Jacko, Salvendy, & Koubek, 1994). Miller (1981) found that short-term memory is a limitation of the increased depth of the hierarchy. His experiment examined four structures (64¹, 2⁶, 4³, and 8²) with a fixed number of target items (64). As depth increased so did response time to select the desired item.

Snowberry, Parkinson & Sisson (1983) replicated Miller’s study by examining the same structures but this time including an initial screening session during which subjects took memory span and visual scanning tests. They found that instead of memory span, visual scanning was predictive of performance, especially in the deepest hierarchies.

Kiger (1984) extended Miller’s research by doing an experiment that provided users with five modes of varying menu designs of 64 end nodes (2⁶, 4³, 8², and 16x4, 4x16). Performance and preference data were collected. The results of the experiment showed that the time and number of errors increased with the depth of the menu structure (The 4x16 structure had the fastest reaction times and the fewer errors). The participants ranked the menus with smaller depth as the most favorable (The 8² structure was favored).

An experiment by Jacko and Salvendy (1996) tested six structures (2², 2³, 2⁶, 8², 8³, and 8⁶) for reaction time, error rates, and subjective preference. They demonstrated that as depth of a computerized, hierarchical menu increased, perceived complexity of the menu increased significantly. Campbell (1988) identified multiple paths, multiple outcomes, conflicting interdependence among paths and uncertain linkages as four characteristics of a complex task. Jacko and Salvendy build on this framework to suggest that these four characteristics are present as depth increases, and the presence of these four characteristics is responsible for the increase in complexity.

Wallace et al. (1987) confirmed that broader, shallower trees (4x3 versus 2x6) produced superior performance, and showed that, when users were stressed, they made 96 percent more errors and took 16 percent longer. The stressor was simply an instruction to work quickly ("It is imperative that you finish the task just as quickly as possible"); the control group received gentler verbal instructions to avoid rushing ("Take your time; there is no rush").

Norman and Chin (1988) fixed the number of levels at four, with 256 target items, and varied the shape of the tree structures. They recommend greater breadth at the root and at the leaves, and added a further encouragement to minimize the total number of menu frames needed so as to increase familiarity.

Zaphiris and Mtei (1997) replicated Kiger’s (1984) structures but this time on the WWW using hyperlinks. Overall their results were in agreement with those of Kiger (1984). They found that of the structures tested (2⁶, 4³, 8², and 16x4, 4x16), the 8² structure was the fastest to search.

Larson & Czerwinski (1998) carried out an experiment using 512 bottom level nodes arranged in three different structures (8x8x8, 32x16, 16x32). Subjects on average completed search tasks faster in the 16x32 hierarchy, second fastest in the 32x16 hierarchy, and slowest in the 8x8x8 hierarchy. Also, on average, subjects tended to be lost least often in the 16x32 hierarchy. Larson & Czerwinski calculated "lostness" through an analysis of the number of unique and total links visited in comparison to the "optimal" path.

One limitation with previous studies on the topic of menu selection has been the use of symmetric hierarchical structures, which do not reflect typical menu structures in real-world applications. Based on a more ecologically valid approach, the present experiment involves an asymmetric hierarchical structure which are common in most practical applications (Norman, 1990).

While previous research has investigated the depth versus breadth effect, it has neglected the type of menu presentation. Specifically, the comparison between expandable indexes with full context and sequential menus with limited context in hierarchical structures on the WWW has not received sufficient attention despite the fact that many designers are employing expandable indexes in current WWW design. It was predicted that expandable indexes will result in superior performance because they provide a full context for the choices at lower levels of the hierarchy. Furthermore, sequential menus are predicted to be less efficient and to result in more backtracking due to loss of context and forgetting on the part of the user.

Participants

Materials

Menu hierarchies were constructed and hosted on the WWW. The bottom level nodes for the various hierarchical structures were taken from the 1996 Annual Survey of Manufactures "Statistics for Industry Groups and Industries" of the United States Bureau of the Census. This survey consisted of a total of 457 bottom level nodes arranged in a specific hierarchy. This structure was used as the basis for designing the different WWW menu organizations used in this study.

Menu structures were generated that varied along two factors: depth of the hierarchy and method of display. Three levels of depth were used: 2, 3, and 4. The method of display used either expandable indexes (Expandable) or sequential menus (Sequential). A factorial design (3 x 2) was used to generate all six possible combinations of levels.

Pages were generated for display in a WWW browser (Netscape 4.0). A top hyperlink bar was also displayed that allowed users to move back to previous pages or to return to the top level of the menu hierarchy. Figure 1 shows a series of displays depicting the expandable index with a menu depth of 4.

Figure 1. Expandable index menu with hierarchical depth of 4. (Hand shows the selection at each level. Arrows show the series of selections but were not shown during actual trials.)

Figure 2 shows an example of the same path through the sequential menu with a depth of 4.

Figure 2. Sequential menu with hierarchical depth of 4. (Hand shows the selection at each level. Arrows show the series of selections but were not shown during actual trials.)

Procedures

Each participant was first told the purpose of the experiment and was then presented with the basic instructions to follow throughout the experiment. Specifically, they were told that we were interested in determining the optimal depth in presenting links on the WWW for expandable indexes and sequential menus that varied the amount of context. In addition, they were given the following general instructions:

1. They were to browse through the assigned pages until they felt comfortable that they found the answer to the assigned task questions.
2. They were to perform all searches as quickly as possible while making as few mouse clicks as possible.
3. They were to navigate through the web pages using only the links present in the pages and not the "back" buttons of the browser.
4. They were instructed not to browse for any other information that they have not been asked to seek until they had finished all assigned tasks.
5. They should read carefully the assigned tasks and instructions presented to them.

After explaining to the participants their tasks, they were given the opportunity to ask questions. All those that decided to participate were asked to read and sign a consent form.

Participants were divided into 3 randomly selected groups (between subjects factor) and assigned to one of the three levels of menu depth. Menu type (Expandable and Sequential) was a within subjects factor and was counterbalanced by presenting different treatment orders. Half of the participants in each group worked with the Expandable Menu first and the Sequential Menu second. The remaining half worked with Sequential Menu first and Expandable Menu second. Each participant performed a total of 10 searches (5 in a Expandable Menu and 5 in a Sequential Menu). The search targets were counterbalanced over the orders and conditions and were selected in such a way that all pages visited had a similar breadth size. The browser height was set such as to just display 20 lines of text. Each trial’s target and hierarchy information was presented to the participant on paper with one target presented per page next to the computer so it was constantly available as a reference. At the start of each trial, the participant was asked to turn the page in order to see the new target and menu type to be used for the next trial.

Three kinds of data were collected for analysis: response times, search efficiency, and subjective ratings. Response times and retrieval efficiency data were collected from the WWW server log. Response time was calculated as the time taken by the participant to browse from the main page to the specific bottom level node. Efficiency of retrieval was calculated as the difference in mouse clicks between the total number of mouse clicks to reach the bottom level node minus the "optimal" number of mouse clicks required to reach the target bottom level node After the participants finished all 10 tasks assigned, subjective preference responses were collected for the Expandable and Sequential Menus by answering on a scale of 1 to 9 the following 3 subjective questions:

1. Which of the two types of menus browsed would you prefer to use?
2. Rate the ease of navigation.
3. Rate your sense of orientation.

Each experimental session lasted around 20 minutes.

Response Times of Information Retrieval

Figure 3 shows that with the Expandable Menu participants completed search tasks fastest in the hierarchy of Depth 2, second fastest in the Depth 3, and slowest in Depth 4. Menu depth had less of an effect for the Sequential Menus with the average response time for Depths 2 and 3 about the same but faster than the response time for Depth 4. Surprisingly, searches were slower for the Expandable Menus than for the Sequential Menus, particularly for hierarchy depths of 3 and 4.

Figure 3: Mean response times as a function of menu type (Expandable or Sequential) and Menu Depth (2, 3, or 4).

The mean response times and corresponding standard deviations for each hierarchy are shown in Table 1.

Table 1: Mean response times for menu type (Expandable or Sequential) and Menu Depth (2, 3, or 4). (Standard deviations are in parentheses)

An analysis of variance of the responses time data indicates a significant effect of hierarchy (F(1, 209) = 13.91, p < .05). A post-hoc analysis using the Bonferroni method showed that the depth 4 hierarchy of the expandable model was significantly slower than any of the other conditions. However there was no significance among the rest of the pairwise comparisons.

Efficiency of Information Retrieval

An analysis of the number of mouse clicks made above and beyond the most direct path the target was performed. Search inefficiency is equal to the total number of mouse clicks to complete each task minus the "optimal" minimum number of clicks. On the average participants in the Depth 2 hierarchies made fewer mouse clicks in the Expandable Menu than in the Sequential Menu. For Depth 3 hierarchies, performance was about the same for the two menu types. However, in the Depth 4 hierarchies performance was drastically worse with the Expandable Menu than with the Sequential Menu. The mean number of mouse clicks in beyond the most efficient path for each hierarchy are graphed in Figure 4 and listed in Table 2 along with corresponding standard deviations. 

Figure 3: Inefficiency of information retrieval as a function of menu type (Expandable or Sequential) and Menu Depth (2, 3, or 4).

Table 2: Inefficiency of Information Retrieval (Standard deviations are shown in parentheses).

An analysis of variance showed that there was no significant effect on accuracy of information retrieval, (F(1,209) = 1.34, p >0.05).

Subjective Ratings and Preference Measures

Table 3: Preference Ratings for the Expandable Menu Over the Sequential Menu. (1 = preference for Sequential Menu. 9 = preference for Expandable Menu. Standard deviations are shown in parentheses.)

We expected that the expandable indexes providing the full context of choices within the hierarchy would result in better performance than sequential menus that provide only limited context. However, this was not the case, particularly for deeper hierarchies where one would expect the context information to be even more beneficial. Surprisingly, tasks were completed faster with Sequential Menus than with Expandable Menus. Efficiency of search data indicated a slightly different pattern, which may help to explain the unexpected result for response time. For Depth 2 hierarchies, the Sequential Menus resulted in slightly more search moves than the Sequential Menu; for Depth 3 performance was equal; but for Depth 4 the Expandable Menu resulted in significantly more search moves than the Sequential Menus.

A possible reason for the poor performance of the Expandable Menu has to do with the problem of long vertical lists on the screen. When the Expandable Menus were expanded in the Depth 4 hierarchies, they were very long and unwieldy. Users opened and closed more levels of the hierarchy and took longer to find the targets because the long indented indexes were hard to scan for context information. Furthermore, context may have been lost as the expanded index scrolled out of view in the limited size of the browser window.

User preferences tended to agree with user performance. However, none of these effects were statistically significant. Users tended to prefer to use Expandable menus for depths of 2 and 3 but when the depth is increased to 4, they chose Sequential Menus as more desirable. Also users found the Expandable menus difficult to navigate (except for Depth 3 hierarchies) and lost their orientation while browsing Expandable menus (except for Depth 3 hierarchy where Expandable and Sequential Menus had equal ratings for sense of orientation).

The results of this experiment replicate results of previous research in the area of depth versus breadth tradeoffs in menu selection. Menu hierarchies should be designed with a minimum depth and maximum breadth if at all possible. Expandable index menus are acceptable only for shallow menu hierarchies or Depth 2 and 3, and should be avoided for deeper hierarchies. If expandable indexes are used, they should be redesigned in such as way as to make relevant, hierarchical, context information clear and available to the user. These designs should avoid hard to follow indentation schemes and long lists that require excessive scrolling of the list in a browser window

Campbell, D. J. (1988). Task complexity: a review and analysis. Academy of Management Review, 13, 40-52.

Jacko, J., & Salvendy, G. (1996). Hierarchical menu design: Breadth, depth, and task complexity. Perceptual and Motor Skills, 82, 1187-1201.

Kiger, J. I. (1984). The depth/breadth tradeoff in the design of menu-driven interfaces. International Journal of Man-Machine Studies, 20, 201-213.

Larson, K. & Czerwinski, M. (1998). Page design: Implications of memory, structure and scent for information retrieval. In the Proceedings of CHI ’98 Human Factors in Computing Systems, Los Angeles, CA, April, 1998, ACM Press, pp. 25-32.

Miller, D. P. (1981). The depth/breadth tradeoff in hierarchical computer menus. Proceedings of the Human Factors Society, 296-200.

Norman, K. (1990). The Psychology of Menu Selection: Designing Cognitive Control of the Human/Computer Interface. Ablex Publishing Corporation, Norwood, NJ.

Norman, K. & Chin, J. (1988). The effect of tree structures on search in a hierarchical menu selection system. Behaviour and Information Technology 7, 51-65.

Snowberry, K., Parkinson, S. R., & Sisson, N. (1983). Computer display menus. Ergonomics, 26, 7, 699-712.

Wallace, D., Anderson, N. & Shneiderman, B. (1987). Time stress effects on two menu selection systems. Proc. Human Factors Society, Thirty-First Annual Meeting (1987), 727-731