Barbara Mirel,
Leif Allmendinger

Visualizing complexity:Getting from here to there in ill-defined problem landscapes

In this paper, we look at experienced problem solvers who are experts in their own domains and who visually model the processes people use when they solve complex problems. Our hope is that improved problem models can inform software development teams and lead to better problem solving software.

We discuss what to model—the interdependent data ordeals, wayfinding, and sense-making activities that make up patterns of inquiry. We propose a model, which describes how experts explore problem landscapes, putting information and their own conclusions together in different ways in order to satisfy contending goals and agendas.

Our long-term goal is to discover how software can better support experts when they solve complex problems. Complex problems differ in kind, not just in degree, from well-defined problems. Examples include an analyst who determines which products you’ll find on the shelves of a large supermarket, or a broker who advises about appropriate investments for you and your family. Such complex problems require distinct support and design approaches.

Complex problems can neither be solved nor supported with linear or pre-defined methods. They have many possible heuristics, indefinite parameters, and ranges of outcomes rather than one single right answer or stopping point. They are also emergent– unexpected possibilities can arise at any point of the problem solving process. Moreover, they have a lot of constraints, some of which invariably conflict, and they have a social dimension– stakeholder pressures locally shape problem solvers’ choices, directions, and goals.

Software has the potential to facilitate this dynamic, open-ended work by allowing experts to explore a problem in ad hoc ways and making reasonable judgments in the midst of uncertainty. Unfortunately, few software applications today realize this potential.

One reason they founder is that problems are inherently difficult to model. A software designer’s view of a problem doesn’t necessarily lead to software that helps experts “get the work right.” Also, in the move from models to conceptual designs, context and complexity often fall through the cracks. The conventional ways in which we represent users’ work do not adequately safeguard against this loss.

In this paper, we propose models for representing problem solving that strive to foster a consistent concentration on context and complexity throughout, so that software designers may create elegant, conceptual designs for situated, open-ended inquiries. We will discuss why models matter and identify some criteria for good complex problem solving models. First, however, to set the stage, we look at what people do when they solve complex problems.

At least once a quarter across the merchandising world, product or category managers decide what items to sell based on multiple contending goals and constraints. They try to find the best mix of products to simultaneously generate high profits, maintain store or dot-com image, satisfy customer behavior and demographics, exploit margins in pricing, respond to supplier incentives, adjust to seasonal variations, and gain advantage over competitors’ strategies. These goals often conflict, and trade-offs are inevitable. Moreover, because product analysts’ considerations are dynamic and vary from one quarter to the next, they continually reassess which product profiles and trade-offs are acceptable and which are not. Insights emerge with each exploratory inquiry, and analysts cannot specify in advance a formulaic process for examining and weighting such relevant data as dollar sales, losses, profits, growth, and traffic.

In studying this class of problem solving, Mirel interviewed and naturalistically observed 27 category (product) managers in 11 work sites as they solved complex product mix problems. These analysts used a pattern of inquiry that they call ‘quadrant analysis’ (Figure 1). This method correlates huge amounts of product performance data and plots them on two measures (the x and y axis, e.g. dollar sales and growth). Display are divided into four cells to distinguish products that are:
1 winners (e.g. high sales and high growth);
2 losers (low sales and low growth);
3 sleepers (high growth and low sales); and
4 opportunities (low growth and high sales, e.g. loss leaders).

For each group of products, product managers analyze and decide which items to keep or drop. They record evolving judgments in a running item list, which changes throughout the many phases of the inquiry. The complexities of even just one phase of this quadrant analysis—deciding what to do with sleepers—come to life in a scenario.

Anya is an experienced category manager in a national grocery chain in charge of shortening and oils in the West region. She is in the midst of deciding the best product mix for this category and has completed her analysis of winners and losers. She now turns to sleepers (see Figure 2).

To analyze this group, she has already (1) accessed and prepared huge amounts of syndicated data from the entire industry on product performance and has merged these data with her proprietary store data. She (2) has plotted the quadrant display correlating sales and growth and (3) has set the dividing lines for the quadrants based on proportions she knows from experience mark appropriate boundaries between winners, losers, sleepers, and opportunities.

She now (4) selects only products in the sleeper quadrant while still keeping the other data visible. Against this context she looks at the lay of the land – the overall performance of sleepers in the category and in each sub-category -- e.g. olive oils, cooking oil, shortening, and pan sprays and microwave sprays. She first examines product performance nationally and then in just her own Western stores. During a later phase of inquiry, she will compare her Western stores with the rest of her regional market. (The syndicated data cover product performance from all grocery stores.)

Moving on and holding the lay of the land in visual memory, she now takes a more detailed look at the data. She (5) narrows in on items belonging to vendors whose combined market shares dominate the category. She sees three vendors dominate and drills down to just their sleepers, still maintaining a view of the rest of the data. Within this subset, she (6) looks only at products that that have been heavily promoted this past quarter. These are candidates for dropping. If heavily promoted products are still ‘iffy’ sleepers, it is foolish for her to throw good money after bad.

At this point, Anya’s progressive and complex drill-down has pushed the query limits of her software’s interactive graphics technology but she does not realize it. The technology cannot go so many levels deep into cumulatively intersecting sets while keeping the data context visible. Consequently, the application returns a graphic display that does not match Anya’s intentions. But Anya does not notice the error until several moves later, after she has already made some decisions about items to drop. Unsure of when, where or why this entangled error occurred, Anya painstakingly retraces and revalidates each step.
After 20 minutes of backtracking and corrections, Anya moves on. She now (7) relates “duplicate” products from preferred suppliers to sleepers she has slated to keep to. She intends to match preferred suppliers’ products to those on her “sleeper-keeper” list and substitute suppliers’ items that are close duplicates.

Suppliers’ incentives and continued loyalty will make up for monetary differences.
In drawing these relationships new data ordeals set in. Anya cannot easily group her preferred suppliers’ products nor cluster only the relevant duplicates into a display that she can readily compare with provisional “sleeper-keepers.” She also wants to but cannot save, annotate, and bring back live views for comparisons later. She does the best she can and finalizes her selection of items to drop and keep by correlating other measures and confirming that they point to the same conclusions.

To conclude this portion of the inquiry, she (8) validates her “drop” choices by running “what-if” scenarios. She creates displays to answer: “What if I drop various products? What can I expect in the bottom line?” The program slows her down and taxes her short term memory. It does not make it easy for her to change her and modify “what-if” scenarios mind on-the-fly based on emerging insights.

Anya experiences a lack of useful support in a number of instances, e.g.:

1. She has trouble integrating current and later phases of inquiry, for example through saved and reactivated views of control over comparisons she needs to make.
2. She lacks the ability to drill deeply into details and see them against the context of the whole.
3. When she discovers -- no thanks to the program -- that she is working with erroneous data for her purposes, she has to significantly disrupt her process of turning quantitative analysis into qualitative judgments to reassess her earlier moves and outcomes.
4. She lacks adequate adaptability for examining her idiosyncratic preferred suppliers’ products and making decisions about substituting duplicates.
5. She cannot readily construct and adapt what-if scenarios to fit her evolving speculations

We propose that if designers model Anya’s inquiry in task landscapes as shown in Figure1 and explained later in Section 4, they may create better support for the needs and demands of complex problem solving. Before getting to to how to model, however, designers first need to know what to model.

Mirel’s research (2003) shows that certain types of complex problems recur in various domains and, for each type -- e.g. product mix problems -- analysts across organizations perform similar patterns of inquiry. Patterns of inquiry are the regularly repeated sets of actions and knowledge that have a successful track record in resolving a class of problems in a specific domain, as does Anya’s pattern of deciding which sleepers to keep and drop. Software designers increasingly employ patterns to represent users’ tasks but, aside from a small body of socio-technical patterns, most patterns describe low-level architecture and software routines or programmatic interface interactions. Because they concentrate on a low level, most patterns highlight interconnected, decomposed logical tasks but do not fully express pragmatic, complex work in context.

Our proposed user models and representational forms, by contrast, focus on pragmatic (socio-technical) level patterns of inquiry. At a pragmatic level, what does Anya do? As illustrated in Figure3, she carries out a number of mainline tasks to progress toward her goals, such as querying only top vendors or setting values for divisions between quadrants that reflect market realities. Mainline tasks are the moves and strategies that alter the structure and content of her analysis and move it forward, Anya also performs many enabling tasks – the sense making and wayfinding actions that support mainline procedures, such as annotating views, keeping thumbnails of saved views visible as reminders for later use, or bringing to bear expert knowledge about the percent market share that counts as vendor dominance in a specific category. Enabling tasks may be invisible (e.g. bringing to bear expert knowledge) or more conscious (e.g. setting cues as reminders), but either way they are vital for domain specialists to do some of their best thinking. They help specialists see how things relate, what to look for, and what choices are most appropriate for mainline procedures. Within patterns of complex inquiry, enabling and mainline tasks interact as an integrated whole. They are chunked – neither discrete nor separable pieces.

In complex problem solving, mainline and enabling task regularities intertwine with local variation. For example, aspects of Anya’s selection are local such as her choices to query preferred suppliers’ duplicate sleepers and the suppliers she examines. These choices are idiosyncratic to her organizational practices and interpersonal relationships.

Patterned and variant dynamics of work are not arbitrary. Their constraints, possibilities, moves, and potential solutions all conform to the real world. Underlying regularities recur and structure specific classes of complex problems but no two inquiries are ever performed in exactly the same way. Because of this “structured openness,” knowledgeable domain specialists who understand the landscapes of their problems should arrive at more effective solutions than those who do not.

These patterned and variant dynamics of inquiry – this structured openness – are difficult to model adequately. It is a challenge to represent patterns of inquiry as the stability onto which problem solvers latch their ad hoc and improvised moves and responses to local constraints and conditions. That is, it is a challenge to capture the interactions, integration, and required coherence among problem solvers’ mainline tasks, enabling tasks, and the contextual conditions and constraints that open some choices and actions for problems solvers while closing others.

Human-centered designing abounds with ways to model users’ work, such as socio-technical patterns, scenarios, use cases, and the various consolidated but separate diagrams of contextual inquiry.

None of these contextual approaches specifically aims to represent the situated, integrated moves and strategies that give coherence to the structured openness of complex problem solving. They capture multiple dimensions and the richness of users’ work but their representational forms too readily encourage a premature leap to turning descriptions into low-level object-oriented conceptual designs without first elegantly designing for users’ holistic investigations. Structured prose and line diagrams, for example, do not adequately highlight the “glue” that gives the coherence to experiences amidst unforeseen constraints and judgments under uncertainty. Therefore, these connective forces get lost. Ecological Interface Designing is a modeling approach that specifically targets complex problem solving but its representations are often too inaccessible for everyday design teams.

Our goal is to represent forms that can be used by designers in any setting that will encourage them to conceptually design for users’ integrated approaches to work before jumping prematurely to feature-by-feature, object-oriented design. Models matter. They affect whether designers build the right work into problem solving software and whether complex problem solvers experience truly useful support. Effective representational forms must encourage designing “just right” coherence into software for users’ complex work. Like Goldilocks, it can neither be too much (overly determined) nor too little (too many overly generic options) or usefulness will be foiled. If software lacks elegant and integrated support for the integrated tasks of complex problem solving, no amount of surface level usability improvements or user experience with an application can lead users to do their work effectively (Sutcliffe et al, 2000).

In thinking about appropriate user models, designers need to be guided by the following criteria for creating representations that capture the integrated quality of indeterminate but coherent work and foster elegant rather than piece-by-piece designs (feature creep). Representational forms should:

1. Orient to users’ pragmatic patterns of inquiry and variation and idiosyncrasy
2. Show the story-like structure of complex investigations, its regular and improvised paths and directions in unfamiliar spaces with uncertain outcomes
3. Capture the synergistic interactions between and within patterns of inquiry
4. Use concrete forms and content that everyday designers can understand and apply.
   

Task landscapes – the mountainscapes in Figure2 – promote a dramatic vision of complex inquiries that visually unify context and actions in one diagram. Mountainscapes depict complex problem solving as explorations across rugged and at times uncharted problem terrains. Problem solvers put together strategic analysis, conventions, intuitions, and extemporaneous choices to move upward through hills and plateaus toward goals. They typically take circuitous routes through “tracts” of tasks, which are portions of landscape that embody integrated paths, chunked task actions, and contextual influences. As problem solvers mutually adapt to and transform paths and surrounding constraints and conditions, they may have little real understanding of the state of the conditions and constraints they will subsequently confront or the consequences their actions will provoke. In the terrain of complex problem solving, analysts have no way of knowing in advance all moves, conditions, constraints or consequences.

Throughout inquiry, some parts of the landscape are much harder to navigate and make sense of than others. The more constraints conflict and conditions are layered, the more rugged and multi-peaked a mountainscape becomes. Sometimes expectedly, sometimes not, problem solvers crisscross the landscape and jump across foothills if they find it advantageous to explore distant but relevant knowledge, to recover from dead ends, or to reinvigorate inquiry when proximate choices are too uninspiring or uncompetitive.

Mountainscape models have the potential to capture human intent and intelligence (the paths) in relation to contending constraints and conditions (the spaces and surrounding climate in task “tracts” affecting how problem solvers conFigurepaths). Visualized as mountainscapes, models of complex inquiry stress relationships among parts and do not readily reduce to linear and rule-based procedures or work flows. The spaces are as important to coherence as the paths.

Anya’s pattern of analyzing and deciding on sleepers, as represented in Figure2, is only one rendition of a mountainscape model. The landscape metaphor, as a type of map, lends itself depicting varying levels of detail through zooming in and out.

As displayed in Figure4, zooming in affords a close up of (a) conditions and constraints confronting problem solvers, (b) moves and paths that problem solvers conFigurethrough this terrain, and (c) salient choices or crossroads occasioned by the interactions of (a) and (b) that critically direct the investigative course. In creating conceptual designs from user models, this detailed perspective is crucial. It helps designers understand the degree of autonomy that software should afford users for different choices and actions under varying conditions and constraints and the flexibility and adaptability they need to design for this use control.

Synergies between inquiry patterns are critical for successful decision-making. Yet long leaps are cognitively expensive. As in the situation represented in Figure6, distant destinations often fit problem solvers’ purposes far better than small step-by-step nearby moves. But after a first long leap, finding and configuring subsequent moves and paths that are equally fit is harder and more costly. In the example represented in Figure6, Anya has to hold her former and new analysis in mind, look at new details and high level comparisons simultaneously, and weigh the risks and gains of various product mix possibilities. Adequate support in problem solving software for these synergistic analyses is crucial since business decision-makers typically choose to engage in low-cost problem solving processes rather than those that are more comprehensive but more cognitively expensive. Today’s software falls short in supporting these synergistic leaps so that better, more comprehensive routes are “affordable.”

These mountainscapes of multiple, integrated inquiry patterns can help designers highlight the data analysis, sense making, and wayfinding that work best for synergistic problem solving processes. They give insight into when it is most efficient, effective and inexpensive for problem solvers’ moves to stay-to-stay close in nearby hills, when leaps are most productive, and how the rhythm of proximate and distant moves may be best orchestrated. Models of problem solving writ large also place domain specialists’ investigative moves in an historical context so that designers can identify how to keep problem solvers oriented to where they started, what information moves and strategies they use at various points, and what meanings they derive and relate within and across sub-goals and patterns.

Modeling complex problem solving as task landscapes establishes the difference between well-structured and open-ended inquiries. Complex problem solving is synergistic, not additive. Stacks of tasks and actions cannot adequately represent it visually. It involves circuitous, recursive and dynamic moves and strategies amidst uncertainty. Models should provide a visual view of problem solvers crisscrossing landscapes of knowledge to put information and meanings together in different ways for the multi-faceted dimensions involved in satisfying contending goals and purposes.

Capturing these dynamics of complex problem solving requires more than modeling task flows and more than separately diagramming various dimensions of work. It requires establishing the problem landscape as an active player in inquiry and decision-making.

Mountainscape forms for representing complex work have several implications for design. Good problem models may lead to tools that reduce cognitive overload. They may also play a critical role in context understanding – helping designers focus on the full range of contexts influencing idiosyncratic and patterned inquiries: the social, organizational, cultural, technological, problem, and cognitive/perceptual contexts. In integrating contexts, actions, choices, and knowledge, mountainscape forms can help designers create software and interface structures that may enable problem solvers to understand the structured openness of their work better so that they may visualize alternatives and solve similar problems more effectively next time.

Finally mountainscapes tell a story while still visually representing these integrated aspects of problem solving. Stories are one of the most fundamental modes of discourse for generating a shared understanding of users’ work among design team members. Mountainscapes can foster this understanding in its form that highlights core interactions that give a story its arc and give users’ experiences their coherence and completeness. Far more deeply than personae or prose scenarios can, this mountainous, exploratory depiction shows the structure of an open-ended inquiry and emphasizes both the patterned and ad hoc and opportunistic experiences involved solving ill-defined problems. With these aspects of inquiry visually rendered, teams may become sensitive throughout design not to let context and complexity fall through the cracks.