University of Southern California Dissertations and Theses

Extraction of preferential probabilities from early stage engineering design team discussion

(USC Thesis Other)

Extraction of preferential probabilities from early stage engineering design team discussion

PDF

Download a page range

Download transcript

Copy asset link

Request this asset

Transcript (if available)

Content

EXTRACTION OF PREFERENTIAL PROBABILITIES

FROM EARLY STAGE ENGINEERING DESIGN TEAM DISCUSSION

by

Haifeng Ji

__________________________________________________________________

A Dissertation Presented to the
FACULTY OF THE GRADUATE SCHOOL
UNIVERSITY OF SOUTHERN CALIFORNIA
In Partial Fulfillment of the
Requirements for the Degree
DOCTOR OF PHILOSOPHY
(INDUSTRIAL AND SYSTEMS ENGINEERING)

December 2008

Copyright 2008 Haifeng Ji
ii
Acknowledgments
I would like to express my sincere gratitude to my advisor and committee chair,
Dr. Maria Yang, for her invaluable guidance and continued support throughout the
journey of my graduate study at the University of Southern California. She has been
an extraordinary mentor and the academia training I have gained is a cherished
experience which will benefit the rest of my life. I am also grateful to my committee
members, Dr. Stephen Lu, Dr. Yan Jin, Dr. Elaine Chew, and Dr. Yong Chen, for their
precious insights and advises during my qualifying proposal and other occasions. I
greatly acknowledge the helpful discussions and the constructive comments from Dr.
Tomonori Honda (Massachusetts Institute of Technology). And I appreciate the
assistance and the cooperation on the research experiment from Mr. Hamid Chabok
(University of Southern California), Dr. Zhihong Shen (University of Southern
California), and Mr. Pavankumar Murali (University of Southern California). Last but
not least, I want to deliver my thanks and love to all my family and friends for their
eternal love and the constant encouragement.
The work described in this research is supported in part by the National
Aeronautics and Space Administration under Cooperative Agreement NNA04CL15A
and also in part by the National Science Foundation under Award DMI-0547629. The
opinions, findings, conclusions and recommendations expressed are those of the
author and do not necessarily reflect the views of the sponsors.
iii
Table of Contents
Acknowledgments ..........................................................................................................ii
List of Tables.................................................................................................................vi
List of Figures..............................................................................................................viii
Abstract.........................................................................................................................xii
Chapter 1 Introduction ................................................................................................1
Chapter 2 Related Work............................................................................................10
2.1 Background...................................................................................................10
2.2 Conceptual Design........................................................................................12
2.2.1 Concept Generation..........................................................................12
2.2.2 Concept Selection.............................................................................13
2.3 Preference Extraction, Aggregation, and Evolution.....................................14
2.3.1 Individual Preference Extraction......................................................14
2.3.2 Group Preference Aggregation.........................................................17
2.3.3 Design Preference Evolution and Design Process............................18
2.4 Language and Text Analysis ........................................................................20
2.4.1 Thematic Analysis and Semantic Analysis ......................................20
2.4.2 Computational Techniques for Text Analysis..................................21
2.4.3 Text Analysis in Design ...................................................................22
2.5 Research Motivation.....................................................................................23
Chapter 3 A Thematic Analysis Approach to Representing the Design Selection
Process 26
3.1 Assumptions .................................................................................................27
3.2 Experimental Study and Data Collection .....................................................27
3.3 Text and Information Analysis Methods......................................................28
3.3.1 Design Selection Evolution ..............................................................28
3.3.2 Design Selection Certainty...............................................................29
3.4 Results ..........................................................................................................30
3.4.1 Design Selection Evolution ..............................................................30
3.4.2 Design Selection Certainty...............................................................34
3.5 Discussions...................................................................................................36
iv
Chapter 4 PPT: A Probabilistic Approach to Extracting Preferential Probabilities
from Discussion Transcripts.........................................................................................38
4.1 Assumptions .................................................................................................40
4.2 Notation ........................................................................................................41
4.3 Preference Transition Model ........................................................................43
4.4 Utterance-Preference Model.........................................................................44
4.5 Preference Calculation..................................................................................47
4.6 Estimation of Hidden Parameters.................................................................51
4.7 Time Intervals...............................................................................................54
4.8 Case Study....................................................................................................55
4.8.1 Case Background..............................................................................55
4.8.2 Data Collection and Method Implementation ..................................56
4.8.3 Results ..............................................................................................59
4.9 Comparisons between Preference Evolution and Frequency Evolution ......63
4.10 Remarks and Discussions.........................................................................66
Chapter 5 Comparative Study between Transcripts and Surveys (1) – Evaluation
with the Logit Model....................................................................................................67
5.1 Existed Models for Conversion....................................................................68
5.2 Logit Model for Converting Ratings to Preferential Probabilities ...............68
5.3 Similarity Measures of Preferential Probabilities from Different Sources ..72
5.3.1 Geometric Distance ..........................................................................72
5.3.2 Cosine Similarity (Dissimilarity) .....................................................75
5.3.3 Pearson Product-moment Correlation Coefficient ...........................76
5.3.4 Spearman Ranks Correlation Coefficient.........................................77
5.4 Experimental Study ......................................................................................79
5.4.1 Team Description and Analysis........................................................80
5.4.2 Experiment Description....................................................................81
5.4.3 Transcript Analysis...........................................................................86
5.4.4 Survey Results..................................................................................93
5.5 Comparisons and Discussions ......................................................................96
5.5.1 Correlations between Survey Ratings and Preferential
Probabilities from Transcripts ......................................................................97
5.5.2 Similarity Comparison between Preferential Probabilities
Converted from Surveys and Extracted from Transcripts............................98
5.6 Remarks and Discussions...........................................................................104
Chapter 6 Comparative Study between Transcripts and Surveys (2) – Evaluation
with PPS: Preferential Probabilities Translated from Surveys under Principle of
Maximum Entropy......................................................................................................106
6.1 Construction of Probabilistic Distribution for Individual Preferences.......107
6.2 Construction of Probabilistic Distribution for Group Preferences .............116
6.3 Simulation of Group’s Preferences ............................................................117
v
6.4 Experimental Study of PPT and PPS..........................................................118
6.4.1 Case Background............................................................................118
6.4.2 Survey Results with PPS ................................................................119
6.4.3 Transcript Analysis with Initialized Preferences............................121
6.5 Comparisons and Discussions ....................................................................123
6.6 Remarks......................................................................................................131
Chapter 7 Conclusions and Future Research ..........................................................133
7.1 Conclusions ................................................................................................133
7.2 Future research ...........................................................................................136
Chapter 8 Contributions ..........................................................................................138
References ..................................................................................................................140
Appendix A: Surveys Used in Coffee Maker Re-design Experiment........................152
Appendix B: A Segment Sample from Discussion Transcripts of Coffee Maker
Re-design (in XML File)............................................................................................154

vi
List of Tables
Table 4.1 Sample Data: Utterances of Alternatives .....................................................56
Table 4.2 Preferential Probabilities of Design Alternatives (the First Iteration) .........58
Table 5.1 Comparison of the New Experiment with the First Example ......................79
Table 5.2 Demographic Information of the Team Members in Coffee Maker
Re-design Project..........................................................................................................81
Table 5.3 Design Information for Carafe Selection ....................................................84
Table 5.4 Design Information for Filter Selection .......................................................85
Table 5.5 Sample Data: Utterances of Alternatives .....................................................87
Table 5.6 Group Preferential Probabilities of Design Alternatives..............................88
Table 5.7 Information Entropy Change on the Component Selection Problem...........91
Table 5.8 Survey Ratings at Time = 10:00 (before Design Process) ...........................93
Table 5.9 Survey Ratings at Time = 20:06...................................................................93
Table 5.10 Survey Ratings at Time = 30:12.................................................................94
Table 5.11 Survey Ratings at Time = 40:00.................................................................94
Table 5.12 Survey Ratings at Time = 48:20.................................................................94
Table 5.13 Group Ratings with Equal Weightings.......................................................95
Table 5.14 Preferential Probabilities with the Logit Model (
λ
=5) on Surveys ...........96
Table 5.15 Preferential Probabilities with the Logit Model (
λ
=20) on Surveys .........96
vii
Table 5.16 Correlation results between PPT and group ratings ...................................98
Table 5.17 Experimental Results of Similarity Comparison between PPT and the
Logit Model................................................................................................................103
Table 6.1 Group Preferential Probabilities from Surveys (PPS)................................120
Table 6.2 Group Preferential Probabilities from Transcripts (PPT)...........................121
Table 6.3 Experimental Results of Similarity Comparison between PPT and PPS ...130

viii
List of Figures
Figure 1.1 Flowchart of the Research Approach..........................................................9
Figure 2.1 General Representation of the Phases of Engineering Design..................10
Figure 2.2 Research Scope .........................................................................................12
Figure 3.1 Word Frequencies for Alternative a
1
of Problem A..................................31
Figure 3.2 Word Frequencies for Alternative a
2
of Problem A..................................31
Figure 3.3 Word Frequencies for Alternative a
3
of Problem A..................................32
Figure 3.4 Word Frequencies for Alternative b
1
of Problem B..................................33
Figure 3.5 Word Frequencies for Alternative b
2
of Problem B..................................34
Figure 3.6 Word Frequencies for Alternative b
3
of Problem B..................................34
Figure 3.7 Information Entropy for Problem A..........................................................35
Figure 3.8 Information Entropy for Problem B..........................................................36
Figure 4.1 Design Process Evolution: Preferential Probabilities of the Three
Alternatives.................................................................................................................60
Figure 4.2 Preferential Probability Ranges of Alternatives a
1
..................................61
Figure 4.3 Preferential Probability Ranges of Alternatives a
2
...................................62
Figure 4.4 Preferential Probability Ranges of Alternatives a
3
...................................62
Figure 4.5 Preferential Probabilities for Alternative b
1
of Problem B .......................64
Figure 4.6 Preferential Probabilities for Alternative b
2
of Problem B .......................65
ix
Figure 4.7 Preferential Probabilities for Alternative b
3
of Problem B .......................65
Figure 4.8 Information Entropy of the Most Preferred Alternative for Problem B ...65
Figure 5.1 Probability Density Function of Logistic Distribution .............................70
Figure 5.2 Cumulative Distribution Function of Logistic Distribution......................70
Figure 5.3 Manhattan Distance, Euclidean Distance and Chebyshev Distance ........75
Figure 5.4 Cosine Similarity between Two Vectors in 2-D Space.............................76
Figure 5.5 Design Process Evolution: Group Preferential Probabilities of the
Three Alternatives for Carafe Selection (Initial Probabilities with Equal
Likelihood) .................................................................................................................89
Figure 5.6 Design Process Evolution: Group Preferential Probabilities of the
Three Alternatives for Filter Selection (Initial Probabilities with Equal
Likelihood) .................................................................................................................90
Figure 5.7 Entropy of Design Selection .....................................................................92
Figure 5.8 Comparison of Group Preferential Probabilities on Glass Carafe ............99
Figure 5.9 Comparison of Group Preferential Probabilities on Stainless-steel
Carafe .........................................................................................................................99
Figure 5.10 Comparison of Group Preferential Probabilities on Plastic Carafe ......100
Figure 5.11 Comparison of Group Preferential Probabilities on Gold Tone Filter..100
Figure 5.12 Comparison of Group Preferential Probabilities on Paper Filter..........100
Figure 5.13 Comparison of Group Preferential Probabilities on Titanium Filter ....101
Figure 6.1 Rating Distribution with Stated Rating=0...............................................110
Figure 6.2 Rating Distribution with Stated Rating=0.2............................................111
x
Figure 6.3 Rating Distribution with Stated Rating=0.5............................................111
Figure 6.4 Rating Distribution with Stated Rating=0.8............................................111
Figure 6.5 Rating Distribution with Stated Rating=1...............................................112
Figure 6.6 Instances of Rating Distribution without Sum Constraint ......................115
Figure 6.7 Instances of Rating Distribution (Sum of Three Sampled Ratings = 1) .115
Figure 6.8 Design Process Evolution: Group Preferential Probabilities of the
Three Alternatives for Carafe Selection (Initial Probabilities with Pre-design
Surveys)....................................................................................................................122
Figure 6.9 Design Process Evolution: Group Preferential Probabilities of the
Three Alternatives for Filter Selection (Initial Probabilities with Pre-design
Surveys)....................................................................................................................123
Figure 6.10 Comparison of Group Preferential Probabilities on Glass Carafe ........124
Figure 6.11 Comparison of Group Preferential Probabilities on Stainless-steel
Carafe .......................................................................................................................125
Figure 6.12 Comparison of Group Preferential Probabilities on Plastic Carafe ......125
Figure 6.13 Comparison of Group Preferential Probabilities on Gold Tone Filter..126
Figure 6.14 Comparison of Group Preferential Probabilities on Paper Filter..........126
Figure 6.15 Comparison of Group Preferential Probabilities on Titanium Filter ....127

xi
Abstract
Activities in the early stage of engineering design typically include the generation
of design choices and selection among these design choices. A key notion in design
alternative selection is that of preference in which a designer or design team assigns
priorities to a set of design choices. However, preferences become more challenging to
assign on both a practical and theoretical level when done by a group of individuals.
Preferences may also be explicitly obtained via surveys or questionnaires in which
designers are asked to rank the choices, rate choice with values, or select a “most-
preferred” choice. However, these methods are typically employed at a single point of
time; therefore, it may not be practical to use surveys to elicit a team’s preference
change and evolution throughout the process.
This research explores the text analysis on the design discussion transcripts and
presents a probabilistic approach for implicitly extracting a projection of aggregated
preference-related information from the transcripts. The approach in this research
graphically represents how likely a choice is to be “most preferred” by a design team
over time. For evaluation purpose, two approaches are established for approximating a
team’s "most preferred" choice in a probabilistic way from surveys of individual team
members. A design selection experiment was conducted to determine possible
correlations between the preferential probabilities estimated from the team’s
discussion and survey ratings explicitly stated by team members. Results suggest that
xii
there are strong correlations between extracted preferential probabilities and team
intents that are stated explicitly, and that the proposed methods can provide a
quantitative way to understand and represent qualitative design information using a
low overhead information extraction method.

1
Chapter 1 Introduction
The early stage of engineering design is a complex process that typically requires
teams of designers and engineers to work together to formulate design problems,
generate possible design solutions, and select appropriate candidate solutions for
additional refinement. A key notion in design selection is that of preference in which a
designer or design team assigns priorities to a set of design alternatives in order to
choose one. A range of methods exist for the overall engineering design process,
including Quality Function Deployment [52], Axiomatic Design [115], and Systematic
Design [95]. Such approaches define principles and guidelines of design process in a
somewhat prescriptive fashion. This research builds on these prescriptive methods
with a descriptive approach to represent design process and design process
information. Specifically, this research examines the phase of design concerned with
design selection. Suppose a design team is presented with a set of possible designs
solutions (A, B, and C) to choose from. There can be any number of reasons a team
might choose design alternative A over alternative B, or C over A, and the team’s
preferred choice may change over the life of a project as the team gathers more
information. Design preference represents a designer’s or design teams’ choices
among a set of alternatives. The research explores the following: “How can what the
teams say in their discussions about these choices be formally analyzed to determine
what choices they prefer?” The extraction of preferential probabilities from such
2
informal design information is a preliminary step towards linking the early, informal
stages of design with more formal, structured design synthesis models.
In engineering design, preference information is mostly represented in the form
of the preference ratings or utilities, which are measures of the satisfaction or
desirability of design alternatives in fuzzy set theory. This research takes a different
perspective on preference information, focusing on preferential probability.
Preferential probability is a probabilistic measure describing how likely one chooses
an alternative as the “most preferred” alternative. For a design team, if one alternative
has a high preferential probability value, it means that the group will likely select that
alternative as the “best” choice among all the alternatives. Preferential probabilities
differ from preference ratings or utilities in the following aspects. The first is that
preferential probabilities describe the likelihood for each alternative to be “most
preferred”, while preference ratings or utilities indicate how much one values an
alternative relative to others, such as desirability, economic value, importance, and etc.
The second is that preferential probability relates the uncertainty of design preferences
drawing on probability theory, while preference ratings or utilities rely on fuzzy set
theory. The third is that the preferential probabilities for all alternatives sum up to 1,
while the sum of the ratings or utilities for all alternatives may not be 1.
This research proposes a probabilistic approach to extract the preference
information in the form of preferential probabilities from the transcripts and evaluates
the extracted preference information with explicit surveys. It is of interest for several
3
reasons. First, it is an implicit method that does not involve direct questioning of
designers to determine their preferences but instead relies on extraction. This
minimizes intrusion on design activities, and therefore increases its usability. Second,
this approach provides a representation of the overall group’s preferential probabilities
without requiring explicit aggregation of group opinion. Group aggregation of
preference has long been a subject of discussion in engineering design and in other
fields, and this approach sidesteps aggregation by treating a group as a single entity
rather than as a collection of individuals. Third, this approach results in a time-based
profile of preferential probabilities that can offer insights into a team’s design choice
evolution. Preferences are generally assessed at only a single point in time, but in fact
a team’s preferences may change throughout the course of a project, and the ability to
chart these changes is a step towards better understanding how teams make
engineering design decisions in early stage engineering design. Fourth, this study
focuses on extraction of preference information and design selection from the raw
discussion data. It provides a powerful tool to harvest information about how team
works on design selection process. Fifth, the quantitative evaluation in this research
provides ways to convert the traditional preference ratings or utilities to preferential
probabilities which are comparable with those from the transcript. The evaluation
ways could enlighten the research on the conversions between these two forms of
preference information.
In this research, the following four questions are explored:
4
What design information can be gained from the design discussion transcripts?
During the design process, design team members communicate and exchange their
ideas in many ways, and studies of the information generated during the process has
been a fruitful area of research. Forms of information examined include logbooks [17],
sketches [108, 136], prototypes [137], and design documentation [84, 114]. However,
these forms of design information represent data that is usually generated only at
intervals during the process (designers may make notes in their logs once or twice a
day, and write reports only at the end of the project). This research examines
transcripts of discussions which provide a representation of design thinking from the
beginning of a project to the end in a somewhat continuous fashion. More importantly,
the language that designers use during design discussion may reflect their design
process and analysis of it can provide valuable insights into the design process [32,
140]. This research question aims to extract design-specific information including
design team preferences for particular design solutions. It examined embedded design
information such as design term selection frequency and selection certainty, and
showed a preliminary way to collect contextual information from the terms
surrounding the design alternative words in the design team discussion.
How can the preference information of a design team be computationally
extracted from their discussions? In the design selection process, the strengths of a
team’s design preferences and the changes in these preferences over time are generally
reflected in designers’ discussion. This research assumes the presence of links
between designers’ preference information (called preference data in this research)
5
and what they say out loud in during team discussion (called utterance data in this
research). Probabilistic models are established to describe the relationships between
designers’ preferences in consecutive time intervals, and the other describes the
relationships between designers’ preferences and designers’ discussion in the same
time interval. The research presented an implicit probabilistic approach that does not
require individual designers to explicitly state their preference. It works to extract
preference information in terms of preferential probabilities from a transcript of design
team discussion as if the group is a single entity The descriptive approach may be a
potential way to link rich but difficult-to-extract informal design information to the
formal design methods.
How does the design preference information of a team evolve over time as the
team changes its priorities on the alternatives? The concept selection process can be
regarded as a process of making trade-off between possible design choices. When
designers make trade-offs, their preferences are influenced as well, and these
preferences can change over time as designers receive new information about their
design constraints. This research introduces a way to represent the design process by
plotting preferential probability changes over time. This study describes the design
process in terms of the preference evolution which is extracted from the analysis of the
designers’ transcribed discussion.
Is the preference information extracted in proposed method consistent with the
actual preferences? In the preliminary study of this research, the preferential
6
probabilities extracted with the proposed approach are mostly consistent with the
qualitative reading of the transcripts. But qualitative reading of the transcripts may not
represent the designers’ actual preferences during design. In order to evaluate the
proposed method, another way of preference extraction is conducted as a baseline. In
this research, surveys, which are stated explicitly by design team members on how
they prefer on each alternative, are used as the frame of reference for evaluation. This
research establishes one conversion method based on the Logit model and another
simulation method under the principle of maximum entropy to translate the traditional
preference ratings to preferential probabilities, which are comparable with the ones
extracted from the transcript. In a new experiment, the preference ratings are elicited
from design questionnaires while the proposed preference extraction method is
employed on the transcribed design discussion data. The results from two sources
(transcripts and surveys) are compared and the proposed preference extraction method
is quantitatively evaluated.
Figure 1.1 gives a flowchart for the research which covers the above four
objectives. The general process centers around a design team as they discuss possible
design choices and make trade-offs between the choices. The design discussion is
audio recorded, transcribed, and time tagged. Text analysis techniques are used to
collect the design specific information, which is called utterance data in this research,
from the transcripts. Utterance data is converted to preference data with the
employment of two models – Preference Transition Model and Utterance-Preference
Model. Initially both the models and the preference data are unknown, EM
7
(Expectation Maximum) algorithm [28] is applied to seek the parameters of the two
models. Finally, the evolution of design preferences is represented graphically,
illustrated by preference strengths at different time intervals. In order to quantitatively
evaluate the preferential probabilities extracted from the transcripts, questionnaires or
surveys are used to elicit preferences from individual designers, which are then
aggregated and converted to group preferential probabilities which are comparable
with those from the transcripts.
This research is organized as follows. Chapter 2 briefly reviews the related
research to this research and clarifies the motivations for conducting this research.
Chapter 3 describes a thematic analysis approach to representing the design selection
process which collects the word frequency information of the key alternatives over the
whole design process. Chapter 4, based on the preliminary study in Chapter 3,
proposes a probabilistic approach to extracting the evolution of preference information
in terms of preferential probabilities from the collection data of word frequencies.
Chapter 5 establishes an approach based on the Logit model for converting the
preference ratings to preferential probabilities which are comparable with those
extracted from the transcripts. An experiment of coffee maker re-design was
conducted for comparative study between the transcript-extracted preferential
probabilities and the preference ratings collected at intervals during the process.
Chapter 6 considers the limitations of the Logit model for engineering design, and
proposes a new probabilistic approach under the principle of maximum entropy to
translate the individual preference ratings to the group preferential probabilities. Both
8
graphical and quantitative comparisons are employed to show the possible consistency
between these two sources (discussion transcripts and surveys). Chapter 7 concludes
the research by reviewing the research questions proposed in Chapter 1 and indicates
some directions in the future research. Chapter 8 summarizes the contributions of this
research.
9

Figure 1.1 Flowchart of the Research Approach
10
Chapter 2 Related Work
2.1 Background
Figure 2.1 shows the flowchart of a typical design process that includes four
phases [95, 99, 124]. The first phase is clarifying the task, which specifies the design
task in detail and collect information about design requirements. The second phase is
conceptual design which determines a principle solution by generating and selecting
solutions to meet the requirements laid out in the first phase. The third phase is
embodiment design, which determines the layout of a technical system using technical
and economic criteria based on the principle solution from the second phase. The last
phase is detail design, which delineates the design of individual parts and production
related documents. Iteration occurs between phases. Important details that inform
decision making are often discovered only after pursuing one activity, causing the
design team to revisit and reconsider previous decisions [77].

Task
Clarifying the
Task
Detail Design
Embodiment
Design
Solution
Conceptual Design
Generation Selection
Iteration

Figure 2.1 General Representation of the Phases of Engineering Design

11
One way of thinking about the early, conceptual stage of design is as a process of
generating and selecting solutions to meet requirements. It corresponds to the first two
phases shown in Figure 2.1. It has been suggested that around 75% of the final
production costs are committed in this stage of design [123], giving the preliminary
design phase an important role in the overall design process. During design selection,
multiple criteria may be considered, and if any of these criteria are at odds, decision
makers may need to make trade-offs among several design alternatives. A key notion
in design alternative selection is that of "preference" in which a designer assigns
priorities to a set of design choices. The design selection process can be regarded as a
process of making trade-offs and assigning priorities to design alternatives.
This research focuses on the conceptual design phase of engineering design,
particularly on concept selection. It applied design-specific text analysis techniques to
extract the design preferences during this design process of this stage. Figure 2.2 gives
a graphic representation of how this study is related to the three main bodies of
research (conceptual design, preference extraction and text analysis).

12
Conceptual
Design
Preference
Extracton
Text
Analysis
Research

Figure 2.2 Research Scope

2.2 Conceptual Design
2.2.1 Concept Generation
During conceptual design, designers generate and selection conceptual
alternatives to meet the requirements which are defined and clarified earlier. This
concept generation process is also known as “populating a design space”. The quality
of the later design is greatly influenced by the quantity and the quality of the design
alternatives populated in this stage. There is a great deal of research in concept
generation focused on algorithmic approaches to formal design synthesis and
configuration, such as Shape Grammars [80], evolutionary and adaptive methods [78],
functional synthesis [21, 82] and value-based design [67]. There are also a lot of
heuristic approaches such as TRIZ (Theory of Inventive Problem Solving) [1, 2],
13
brainstorming [71, 90, 91], transformation method [113], and language stimuli
approach [22].
2.2.2 Concept Selection
Concept selection is a process for narrowing the design space to select a concept
or concepts for later design and development. This concept selection process is a
convergent step while the concept generation process could be regarded as a divergent
step [82, 99]. During this convergent step, designers optimize the design objectives by
comparing and selecting the design alternatives which are generated earlier. Several
approaches have been proposed for this stage, including Decision Based Design [53],
Analytic Hierarchy Process [104], Pugh Charts [99], optimal design [96], robust
design [97], and game theoretic approaches [79].
One formal view of conceptual design is as a result to be optimized. The
optimization of conceptual design involves trading off design attributes between
different design alternatives and design parameter values. These trade-offs occur by
comparing all relevant performances of particular design with others. Methods for
formalizing design trade-offs include Utility Theory originally developed by von
Neumann and Morgenstern [60, 70, 117-119, 125] and the Method of Imprecision
(MoI) proposed by Otto, Antonsson, and Wood [92, 135].
Both of these methods are based on the designers’ preferences for multiple
criteria of a set of design alternatives, but there are several differences between Utility
Theory and Method of Imprecision. The main difference is the axiom for aggregation
14
scheme for preferences (utilities) for performances of design. Utility theory is
originally developed for economic decision making and not engineering design trade-
offs, making the overall utility to be restricted to be additive given by Archimedean
property. On the other hand, Method of Imprecision has the property of annihilation
such that if one of preference is 0, the overall preference will be 0. Thus, one
performance failure cannot be compensated by other performance improvement for
Method of Imprecision. Both theories use the same individual performance preference
measures before aggregation. Thus, it is important for both approaches to obtain
individual performance preferences which this research is trying to achieve.
2.3 Preference Extraction, Aggregation, and Evolution
2.3.1 Individual Preference Extraction
Design preference is a qualitative or quantitative measure that represents the
designer’s choices among a set of design alternatives based on a certain combination
of criteria. Design preferences can be expressed as an ordinal, or in a quantitative way,
commonly in a scale from 0 to 1 where the higher value represents more of a
designer’s favor.
In formal design approaches, accurately modeling the strength of preferences is
very important in the design decision making process. One common way to determine
a designer’s preference is to explicitly ask them via surveys or questionnaires. The
lottery method [74] is a classical way to elicit preferences in a quantitative way in
which alternatives are preferentially scaled from 0 to 1 based on a questionnaire. First,
15
the designer is asked to identify the most preferred alternative x
max
and the least
preferred alternative x
min
over the alternative collection X, and these two alternatives
are given the numerical ranks 1 and 0. Then, for every other alternative x
k
in X, a
lottery question is asked [53, 93]:
“On a scale from 0 to 1, what is your numerical belief μ that you are indifferent
between receiving the following two options:
x
k
for sure;
x
max
with certainty μ and x
min
with certainty 1- μ.”
This method provides a numerical rank over the alternative collection, but
inconsistency can be a problem when using the lottery method to assess utilities [7, 23,
24, 27]. In order to ensure the consistency, pair-wise comparisons are made to
determine the relative performance of the alternatives in terms of each individual
criterion. This approach is often used in the Analytical Hierarchy Process [104] as a
decision making support tool. Other quantitative pair-wise comparison approaches
include: Wang [129], who employs a fuzzy preference relationship to discriminate
three preference models for design evaluation, and Li and Jin [81] who apply this
fuzzy preference relationship to select alternatives. All of these methods rely on
directly asking designers for their preferences, or assuming some value for them. Wan
and Krishnamurty [128] proposed a learning-based preference modeling for
engineering decision-making, which could improve formulating utility functions by
16
integrating a deductive reasoning-based on designer’s outcome ranking in a lottery
questions-based elicitation process.
Probabilistic ways to extract preferences have also been considered in research.
Two methods based on factorial design and statistics are Conjoint Analysis [48] and
Discrete Choice Analysis [9, 55]. Conjoint Analysis groups attributes and requires
respondents to make tradeoffs to reveal their preferences. Discrete Choice Analysis is
a statistical technique that identifies choice patterns individual respondents make
between different alternatives. It originated in transportation engineering, and it builds
upon design of experiments, formal data collection, and different models such as
Linear Model, Probit Model, Logit Model, Arctan Probability Model, Truncated
Exponential Model, and Generalized Extreme Value Model [9, 12] to predict the
probability that an alternative is selected from a set of alternatives. In engineering
design, Wassenaar and Chen employ discrete choice analysis in a Decision-Based
Design framework to determine preferred alternatives [131]. All of these probabilistic
ways for extracting preferences are also based on explicit surveys of respondents.
Another, lower overhead approach is to extract preferences from a designer’s
actions. Collaborative filtering [73] is based on the assumption that individuals with
similar profiles gravitate to the same choices. Collaborative filtering approaches are
often classified as memory-based approaches and model-based approaches [130].
Memory-based collaborative filtering stores all rating information as-is in the memory,
and includes two methods: user-based methods [16, 56] and item-based methods [29,
17
105]. Model-based collaborative filtering uses training examples to generate a model
that predicts the preferences for the new items. Different models, such as decision
trees [16], latent semantic models [58], and factor analysis models [20], are used in the
model-based filtering research.
Collaborative filtering has been used in many commercial applications,
particularly in sales and marketing. However, collaborative filtering requires a
relatively large number of individual opinions to be effective, more than is typically
on a small design team.
This research introduces a new approach to extract preference information in
terms of preferential probabilities from the transcribed discussion of design teams that
does not require individual designers to explicitly state their preferences as in other
methods. It assumes that designers’ preferences are somewhat related to what
designers discuss during the design process, and this kind of relationship can be
modeled in probabilistic ways. In this way, the process of eliciting preferential
probabilities will be less disruptive to the team, and may potentially be more accurate.
2.3.2 Group Preference Aggregation
Designer preferences are often considered for a whole team rather than the
individual member alone, and several approaches exist to aggregate group preferences.
Arrow’s Theorem [3, 4] demonstrates that there is no guarantee of consistency in a
group. It is reasonable that the team members may have different preferences, and it is
more important to consider the group preference rather than the individual preferences
18
for the group design. Dym, Wood and Scott [39] discuss Pair-wise Comparison Charts
(PCC) to aggregate individual orderings of design team members into “group
decision”-like voting. Keeney uses cardinal utility functions to accumulate group
preferences [69], while Bask and Saaty [8] apply Analytical Hierarchy Process and
pair-wise approach to aggregate the group preferences, assuming each group
member’s opinion is equally important. Jabeur, Martel et al.[62], and See and Lewis
[107] go one step further and assume unequal weights on the preferences of the group
members, although this sometimes requires prior knowledge of weighting factors,
such as a member’s technical expertise.
This research takes a different approach that does not require prior knowledge of
an individual’s background or ways to formulate unequal weights. Instead, the
preferences of a team are considered from the point of view of group discussion, and
group preference information is extracted directly from the text analysis of a single
transcript of the design team’s discussion to obtain a projection of aggregated design
team preferences. The design team is treated as a single entity.
2.3.3 Design Preference Evolution and Design Process
Much past research in modeling design preferences has focused on representing
preferences at a single point in time. In fact, preferences can change over time. Such
changes may be due to the introduction of new information or changes in the state of
the decision-making entity [83], or the dynamic nature of the selection process in a
group setting [14]. Furthermore, design may be considered a continuous process of
19
making iterative trade-offs, particularly as teams better understand their objectives,
requirements, and constraints change. In this sense, the design process can be thought
of as an evolution of preferences.
One of the necessary elements for assessing design preference evolution is a
sufficiently rich source of design information to draw upon. The language that
designers use during design discussion may reflect their design process and analysis of
it can provide valuable insights into the design process [32, 140]. Preference
information may be embedded in the qualitative design information that designers and
teams generate, such as logbooks [17], sketches [108, 136], prototypes [137], and
design documentation [84, 114]. These forms of design information represent data that
is generated only at selected intervals during design rather than continuously. For
example, designers may make notes in their logs once or twice a day, and write reports
only at the end of the project. These forms of information may only record the parts of
the design process that a particular designer is willing to commit to paper.
Furthermore, logbooks may reflect only the opinions of a single designer rather than
the overall thinking of an entire design team.
Teams are a common work group unit in engineering practice, and the
deliberation that goes on within a team can have major influence on the choices that
are made for a design. This work looks to verbal design team discussion as an
information source because it includes the opinions of multiple team members. In a
20
group setting, there is also back-and-forth discussion on issues, providing an
opportunity to elicit design rationale explicitly from team members.
Several researchers have examined the design process over time through surveys
[17], coding of design journals [64], “story telling” [114], word frequencies and
information certainty analysis [138], and language aggregation and language
accumulation [34]. This study describes the design process in terms of the preference
evolution which is extracted from the analysis of the designers’ transcribed discussion.
2.4 Language and Text Analysis
2.4.1 Thematic Analysis and Semantic Analysis
Text analysis (also called textual analysis or content analysis) [10, 59, 110] is a
technique originally used in social sciences to describe and analyze the manifest
content of communication. The method of text analysis enables the researcher to
include large amounts of textual information and systematically identify its properties,
e.g. the frequencies of most used keywords, by detecting the more important structures
of its communication content. And there are two major groups of text analysis
methods [102]: thematic analysis and semantic analysis.
Thematic analysis measures occurrences of themes or concepts. The
measurements which are used depend on the questions which are to addressed. As this
method is focused on the themes of the text, it does not fully consider context from
which the text was generated, which makes the analysis inconsistent with the original
21
meaning in the text. Danielson and Lasorsa [26] illustrated a detailed thematic content
analysis which examines theme prevalence during 100 years of front page news in The
New York Times and The Los Angeles Times. They statistically collected and plotted
the frequencies of agricultural words, human element words, etc. The analysis reveals
major social and political changes in American society. The New York Times website
1

also transcribed President Bush’s State of the Union addresses from 2001 to 2007.
With an average of 5,000 words in each address, The New York Times gives a
frequency dot plots with which one can find the political changes in recently year and
may even predict the further politics.
Semantic analysis examines the sentences or clauses in which themes are
contextually interrelated. In this method, not only themes but grammatical relations
among themes are encoded in semantic grammars [18, 101] or functional grammars
[30, 50]. Semantic analysis methods had been widely used in social science to retrieve
grievances [86, 111], labor disputes [44, 45], individual’s psychological states [47],
and etc.
2.4.2 Computational Techniques for Text Analysis
There is a great deal of work in computer science that examines ways of
extracting and categorizing collections of text documents. Riloff [100] proposed
algorithms for automatic text categorization and domain-specific dictionaries.
Kitamura et al [72] described a semantic network for representing the functionality of

1
http://www.nytimes.com/ref/washington/20070123_STATEOFUNION.html
22
engineering devices. Hearst [54] examined approaches to generate useful groupings.
Word similarity and semantic analysis also play a key role in text analysis. Many
statistical word similarity measurements have been formulated [38, 49, 76, 78, 120],
and one well-known application tool is WordNet [87], which is an online lexical
reference system used in semantic analysis and information analysis across many
domains. While these approaches are important and useful for informing the work
described in this research, they are less relevant to representing evolution specifically
for design.
2.4.3 Text Analysis in Design
Language-based communication in design includes design documentation and
written transcriptions of design discourse produced during the design process.
Ericsson and Simon [41] employed verbal protocol analysis (“think aloud”) as a
source of data about how designers work, and they argued that such verbalization
would not alter the thought process because it drew on short-term memory. Verbal
protocols have been widely used in engineering design studies, such as engineering
student design processes documentation [5, 6], design team interaction [32], and
design enactment [34]. A big challenge to studying unstructured language in design is
the ambiguity of language, including imprecise and informal language use, word
synonymy and polysemy, and etc. Some research has been done on this aspect to
resolve this design language problem, such as building and employing design-specific
23
thesauri [140], design ontologies [134], latent semantic approach [31] and lexical
chain analysis [36].
Several researchers have examined design process through surveys [17] and
coding of design journals [64]. This study takes a slightly different approach to
assessing design process by analyzing the text generated during design. Dong [31, 35]
performed Latent Semantic Analysis (LSA) on design documentation and verbal
communication in design teams to express the coherence of design team documents.
Correlations were found between more coherent design documentation and better
design outcomes. Simpson, et al [112] explored the relationship between the design
freedom and the information certainty in the early stage of the design process. Chiu
and Shu [22] linked design process language using WordNet as a framework, and built
corresponding lexical relationships. Perhaps most closely related to the work described
in this research are the efforts of Song, et al [114], who viewed engineering design as
a process of “story telling” and depicted the coherence evolution of the design team in
the design process. This research builds on previous design text analysis work, and
differs in that it focuses on transcripts of designers rather than with the documentation
explicitly created by designers, and also uses both text analysis and preference
extraction to understand design choice evolution.
2.5 Research Motivation
These related bodies of research contribute a great deal to understanding and
improving the early stage design, but there are no models connecting measures to the
24
design process over time to provide insight into the relationship between design
processes and design outcomes. The design selection process can be regarded as a
process of comparing and choosing the “best” solutions, so it can be modeled as a
process of design preference changes. In order to model the preference evolution, the
changes of the preferences should be captured over time. This is the first motivation of
this research.
The second motivation comes from the requirements of formal design synthesis
or design decision making. There are several methods for formal design synthesis or
design decision making, including Pugh charts [99], Quality Function Deployment
(QFD) [52], Decision-based Design (DBD) [53], and the Method of Imprecision (MoI)
[92, 135]. Such approaches require individuals or teams to state their preferences
explicitly. In particular, when a team assigns preferences, then the team must arrive at
a single set of preferences. This process often involves group discussion which can be
a highly constructive way to understand design issues. At the same time, it can be
difficult to elicit this type of shared group opinion in a formal fashion. A common way
to obtain preferences from a group is through surveys or questionnaires in which
respondents are asked to rank their choices [15, 94], rate choices with values [70, 106,
117, 119, 125], or select a “most preferred” choice [19, 42, 51, 83]. However, it is not
always practical to use such methods in an engineering workplace environment as
their administration involves considerable overhead and may be disruptive.
Furthermore, such methods are typically employed only at a single point in time, but
in fact a team’s preferences may change throughout the course of a project. This
25
research aims to design a new implicit alternative method which could compensate the
flaws of the traditional questionnaire or survey method.

26
Chapter 3 A Thematic Analysis Approach to Representing the
Design Selection Process
This chapter describes a text-based thematic analysis method to describe the
design selection process. This study uses word frequencies to represent the evolution
of the selection of design alternatives, and information entropy to model the
“certainty” of these design choices over time. The research described in this section
examines the transcribed discussions of a team designing a large-scale system
architecture in an industry setting.
In early stage design, teams typically generate a number of initial concepts and
then choose among them, often iterating on the process. What might this process of
generation and selection look like graphically speaking? Consider the number of ideas
generated during the conceptual design process. It might be expected that the total
number of ideas would increase during the concept generation phase and decrease in
the concept selection phase as less desirable ideas are weeded out. In the case of the
system architecture problem presented in the preliminary case study, the focus is on
the trade-offs among a smaller number of alternatives rather than on concept
generation. It would be anticipated that the volume of concepts would oscillate over
time as decisions and trade-offs are made.
Often during conceptual design, there is ambiguity as to whether a design choice
is completely settled or “final.” Ambiguity may be valuable in the early stages of
design because it can present design freedom. At the same time, understanding the
27
certainty of design choice can help a design progress more efficiently. To gain insight
into the certainty of design selections, this section uses information entropy to
represent selection certainty. It is expected that design selections become more and
more certain as discussions progress over the course of a design process.
3.1 Assumptions
This section makes several simplifying assumptions:
Assumption 1: Embedded in group discussion is information sufficient to reflect
group preference. During the design process, what designers say to each other
generally corresponds with what they think. This is also an implicit assumption of
protocol studies of designers [25].
Assumption 2: All major design alternatives for a concept selection problem are
largely known a priori. While this may not be true for novel design problems, it is a
reasonable assumption for incremental or re-design problems in which many design
alternatives under consideration are ones that have been examined in the past.
Assumption 3: The more frequently a designer says an alternative, the higher the
likelihood it will be chosen for the design, and this relationship between word
frequency and selection probability is linear.
3.2 Experimental Study and Data Collection
The case used for the thematic analysis in this section is a real-world design team
working on the design of a large-scale space system architecture. This design team
28
was composed of 17 experienced scientists and engineers of different disciplines
working together in a co-located, highly concurrent setting. The team had collaborated
on several similar projects in the past. The project took place over three 3-hour
sessions spaced out over several weeks. This research focuses on the audio-recorded
utterances of one member of the team as he explained his design decision-making
process in detail to a novice member of his team. This recording was transcribed into a
text document of approximately 28,000 words. All data was time coded. In the
transcript, the primary team member talked nearly 85% of the time, and four other
members made up the remainder.
3.3 Text and Information Analysis Methods
3.3.1 Design Selection Evolution
The design project included two design component selection problems, A and B.
For each selection problem, there were 3 alternative candidates. Let a
1
, a
2
, a
3
be the
alternative candidates to be chosen for Problem A, and b
1
, b
2
, b
3
be the alternative
candidates to be chosen for B. Much of the design process involved making trade-offs
among all the alternative candidates before finally selecting one of three candidates for
each design component selection problem (A and B). The word occurrences of all six
alternatives were collected and plotted over time. These frequencies provided a sense
of how much a concept was discussed, and the frequencies of all 3 terms together gave
an idea of overall evolution of design choices.
29
3.3.2 Design Selection Certainty
For each design alternative in a discussion, the probability that each alternative
will be chosen can be calculated based on Assumption 3. Suppose in a selection
problem with n alternatives, designers prefer to select the alternative j (j = 1…n) with
a probability P
ij
in the i
th
time interval, then P
ij
can be inferred using maximum
likelihood estimation [43] as in Equation (3.1):

1
ij
ij n
ij
j
f
P
f
=
=
∑
(3.1)

where f
ij
is the number of occurrences of the alternative j in the time interval i. With
the above probability of occurrences of alternatives, the alternative j is the most likely
to occur f
ij
times if the other alternatives of the same design selection problem occur
with certain frequencies.
In a typical design selection process, it might be expected that a choice starts out
with relatively low certainty, and becomes more certain towards the end of a project.
Information entropy [109] is a measurement of the amount of uncertainty about an
event associated with a given probability distribution. The lower the value of
information entropy, the more certain the design choice is. Assumption 2 guarantees
the completeness of the alternatives in the design selection problem. Let K
i
be the
selection variable in the i
th
time interval, whose value can be chosen from the
30
alternative range {1, 2, …, n} for a component selection problem with n alternatives.
The probability of each alternative of being chosen is known from Equation (3.1), then
the entropy value of the selection variable K
i
can be calculated as in Equation (3.2).

2
1
() log
n
iij ij
j
entropy K P P
=
=−
∑
(3.2)

Equation (3.2) explains how “certain” one design choice is in the i
th
time interval.
Entropy values plotted over time illustrate the changing certainty of the design process.
This plot can suggest if designers are sure of the alternative they have chosen already
or have still not made a final decision.
3.4 Results
3.4.1 Design Selection Evolution
The word occurrences of the six alternatives for the two selection problems in
intervals were collected. Figures 3.1-3.3 plot the frequencies of the three alternatives
of Problem A at intervals of 10 minutes. The dotted vertical lines mark three sessions
in which the design process took place. Figure 3.1 shows that Alternative a
1
was
discussed in the first period and then again in the middle of the second period, but it
was not mentioned in the third period. Alternatives a
2
and a
3
were only mentioned at
the beginning of the first period (Figures 3.2 and 3.3). Comparing Figures 3.1-3.3 for
Problem A shows that mentions of Alternative a
1
dominated the entire design process.
31
This is consistent with a qualitative reading of the transcript which showed that, after
one alternative was “selected” in the first session, only the selected alternative was
mentioned in the late first session and the second session for deciding the alternative
for another component selection problem.

0
2
4
6
8
1 3 5 7 9 1113151719 21 232527 29313335 373941 43 454749
Time (in 10 minute intervals)
word occurrences

Figure 3.1 Word Frequencies for Alternative a
1
of Problem A

0
2
4
6
8
1 3 5 7 9 1113151719 21 232527 29313335 37394143 454749
Time (in 10 minute intervals)
word occurrences

Figure 3.2 Word Frequencies for Alternative a
2
of Problem A
32

0
2
4
6
8
1 3579 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 49
Time (in 10 minute intervals)
word occurrences

Figure 3.3 Word Frequencies for Alternative a
3
of Problem A

Figures 3.4-3.6 depict the frequency evolution of the three alternatives of
Problem B. Figure 3.4 shows how Alternative b
1
of Problem B was mentioned often
throughout the first period, briefly at the start of the second, and then again throughout
the third. This is consistent with a qualitative reading of the transcript which showed
that the alternative was “selected” at the end of the first interval, only to be
reconsidered in the third. Figure 3.5 complements Figure 3.4 and shows that, in the
second stage, Alternative b
2
was discussed after Alternative b
1
had already been
chosen. Figure 3.6 shows that Alternative b
3
was discussed constantly throughout the
project. From comparison of Figure 3.4-3.6, it can be seen that Alternative b
3
was
discussed more frequently than Alternatives b
1
and b
2
throughout the design process in
Problem B. Taken together, these basic frequency plots show graphically when and
how alternatives are being considered with respect to other design choices. These plots
33
were surprisingly consistent with qualitative assessments of the transcripts, and
seemed to accurately reflect the general decision-making process of the designers.
Interestingly, one of the findings of Yang [139] was that design language evolves
rapidly over time because of the inherently changing nature of conceptual design
process. For example, at the beginning of a project, a designer may refer to a
component more generically (a “thing”) and then become more specific as design and
function become clearer. However, the same pattern of changing terminology was not
found here, in part because large scale re-design projects tend be based on well-
established “legacy” projects, and the language used to describe the projects likewise
tends to be fairly fixed.

0
2
4
6
8
10
1 3 5 7 9 11 131517 1921 23 25 2729 3133 3537 3941 4345 4749
Time (in 10 minute intervals)
word occurrences

Figure 3.4 Word Frequencies for Alternative b
1
of Problem B

34
0
2
4
6
8
10
1 3 5 7 9 1113 1517 19 2123 2527 29 3133 3537 39 4143 4547 49
Time (in 10 minute intervals)
word occurrences

Figure 3.5 Word Frequencies for Alternative b
2
of Problem B

0
2
4
6
8
10
1 3 5 7 9 111315171921 2325272931333537 394143454749
Time (in 10 minute intervals)
word occurrences

Figure 3.6 Word Frequencies for Alternative b
3
of Problem B

3.4.2 Design Selection Certainty
The selection probabilities of all six alternatives and information entropies of the
two design choices at all time intervals were calculated according to Equation (3.1)
and Equation (3.2). Figure 3.7 shows that entropy value of Problem A is almost 0
35
except at the beginning of the design process. It means that the designers only
discussed this component selection problem early on and were perhaps more certain
the alternative should be chosen for this selection problem from the later part of the
first period until the end. From the combination of Figure 3.7 and Figures 3.1-3.3, it
could be predicted that Alternative a
1
would be chosen for Problem A, which was
validated by examining the original transcript. Figure 3.8 shows the information
entropies of Problem B plotted over time. It suggests that the team actively discussed
the alternatives in the first period and perhaps agreed on an alternative in the early
stage of the second period where the entropy value is 0, but then diverged again in the
third period. This conjecture is partially validated by correspondences with Figures
3.4-3.6, and also by a qualitative reading of the original transcript.

0
0.2
0.4
0.6
0.8
1
1.2
1.4
1.6
1 3 5 7 9 111315 1719 2123 2527 2931 3335 3739 4143 4547 49
Time (in 10 minute intervals)
Information Entropy

Figure 3.7 Information Entropy for Problem A

36
0
0.2
0.4
0.6
0.8
1
1.2
1.4
1.6
1 3 5 7 9 11 1315171921 2325272931333537394143454749
Time (in 10 minute intervals)
Information Entropy

Figure 3.8 Information Entropy for Problem B

3.5 Discussions
The methodology presented in this section suggests a way to model the design
selection process through word frequency analysis and entropy analysis, and may lead
to a novel way to understand the nature of design choices over time. The preliminary
study in this section gives two case examples that show somewhat different results
regarding the evolution of design alternatives and the evolution of design selection
certainty. It was expected that design alternative choices would oscillate in a large-
scale system design problem, and this was true for some design alternatives but not for
others. It was anticipated that information entropy would decrease overall with the
progress of the design process but the results suggest that patterns are more complex.
While this pattern was true for Problem A, the results for Problem B show that
decisions were settled and then became open again for discussion. However, the
design process of Problem B did not seem to be well resolved in Figure 3.8, as the
37
alternative choice was not certain even at the end of the third period. The results of
this preliminary study indicate that this approach may be useful in illustrating in detail
the progress of early stage design process, but need to be validated and refined on a
broader range of design problems.
There are several scenarios in which the methods described in this section may
not apply. The first is when the alternatives are unknown. This is often true in the case
of a novel design, when the alternatives are not planned in advance. Another exception
is when the selection range of the design variable is continuous. An example of a
continuous variable might be the diameter of a component. There are infinite
alternatives in the case when the design variable is continuous, so it is impossible to
collect the information of all alternatives.
In this section, what designers think is strongly linearly related with what
designers talk about the alternatives. In the next section, the third assumption is
modified to more closely reflect real-world problems. Instead of assuming a directly
linear relationship, a more complex relationship can be modeled between what
designers say and what designers think. Two models are built in the next section to
describe this relationship. Applying the models to design documentation or transcripts
will provide a more accurate model of the designers’ selection process and certainty.

38
Chapter 4 PPT: A Probabilistic Approach to Extracting Preferential
Probabilities from Discussion Transcripts
This chapter presents an implicit probabilistic approach for extracting a
projection of aggregated design team preference information from design team
discussion as if the team is a single entity. It further takes into consideration how the
design preference information of a team can evolve over time as the team changes its
priorities based on new design information. Two initial models are given for
representing the most probable and preferred design alternative from the transcripts of
design team discussion, and for predicting how preferences might change from one
time interval to the next. This section examines three aspects of preferences in design
teams: design preference extraction, projecting an aggregation and understanding
preference evolution over the life of a project, and presents a case example to illustrate
its approaches. For the sake of compactness, the probabilistic approached in this
chapter is named with PPT (Preferential Probabilities from Transcripts).
Figure 1.1 shows that the preference data are acquired from the utterance data of
the transcript. But the linkage between them is not as easy as the other steps in Figure
1.1 because the models and the parameters of the models linking the utterances and
preferences are unknown. The basic procedures of this approach to resolve this
problem are shown as follows:
39
1. Collect word occurrences of all design alternatives in a transcript of a design
team’s discussion. The collection of word occurrences is called utterance data. In this
step, variations of specific terms (synonyms) that represent the same alternative are
also included as occurrences.
2. Build a preference transition model to describe the relationship between
preferences in two consecutive time intervals, along with an utterance-preference
model to describe the relationship between what designers say and what designers
prefer within the same time interval. The parameters of the two models are unknown
(details in Sections 4.3 and 4.4).
3. Assign reasonable initial values to the parameters of these two models.
4. Apply both models to a transcript to predict preference data. The preference
data will be used to describe the evolution of preference information over the design
process (details in Section 4.5).
5. Update the parameters of these two models using a traditional Expectation-
Maximum (EM) algorithm [28] on the predicted preference data and the given
utterance data (details in Section 4.6).
6. Repeat steps 4 and 5 until there is convergence on the hidden parameters of the
models. Parameters converge because the EM algorithm is guaranteed to improve the
probability of the occurrences of the utterance data at each iteration [13].
40
4.1 Assumptions
In this section, five assumptions are made with regard to the preferential
probabilities extracted from group transcripts. The first two are the same as the first
two in Section 3. And another three new assumptions (Assumptions 3, 4 and 5) are
appended as follows.
Assumption 3: An entire discussion can be divided into time intervals during
which the designers’ preference are assumed to be unchanged. A change in preference
can only occur between consecutive time intervals. This assumption helps to divide
the whole design process into intervals, in order to describe the preference change of
the design team during the design process. The ways of division are described in
Section 4.7.
Assumption 4: what is most preferred in a time interval is related to what was
most preferred in the previous time interval. This relationship can be represented
probabilistically, and describes how likely the design team would change the “most
preferred” alternative.
Assumption 5: Designers tend to talk positively more about the design alternative
they prefer more and negatively about the design alternative they prefer less. Within
the same time interval, how often an alternative is mentioned positively or negatively
is linked to how much it is preferred, and this can be represented probabilistically.
This assumption is an extension of Assumption 1. Probabilities are used here because
of the stochastic uncertainty in the group discussion. It is possible for designers to
41
mention the less-preferred alternative positively interlacing with the negative meaning,
or mention the most-preferred alternative negatively interlacing with the positive
meaning. This probabilistic relation considers that occasionally people do not mean
what they say [11], but people still tend to speak out what they really mean.
Probabilistic models describe how likely the team talks about the alternative they
prefer the most.
4.2 Notation
The following explains some symbols which are used in the later mathematic
formulations.
N: total number of alternatives in the studied design selection problem
T: total number of time intervals over the whole design process
i, j: the index to represent different time intervals in the design process
m, n, k: the index to represent different alternatives in the studied design selection
problem
r: the index to represent the different iterations of the calculation process
m
a
: the mth alternative of of the studied design selection problem in the design
process
A: the set of all alternatives, i.e. A={
12
, ,...,
N
aa a
}
42
i
π
: the alternative which designers prefer to all other alternatives in Time Interval
i, i.e., the most-preferred alternative in Time Interval i
i
ε
: the alternative which designers utter at sometime during Time Interval i of the
design process
i
σ
: the sequence of the utterances of design alternatives in Time Interval i. e.g., if
in Time Interval 2, the design alternatives are uttered as
2 2 1 1 12 31 1 3
,, , , ,, , , , aaa a a aa a a a
in
the designers’ transcribed discussion , then
2
σ
= {
2 2 1 1 12 31 1 3
,, , , ,, , , , aaa a a aa a a a
}
()
im
Pa π =
: the probability that designers prefer Alternative
m
a
to all other
alternatives (i.e., Alternative
m
a
is most preferred) in Time Interval i. If the preference
value of Alternative
m
a
in Time Interval i is represented by
()
im
a μ
on a scale from
zero to one, then
im
a π =
is equivalent to
() ( ) 1
im in
a a for all n N μμ≥≤≤

(| )
im j n
Pa a ππ ==
: the probability that designers prefer Alternative
m
a
to all
other alternatives in Time Interval i, given that designers prefer Alternative
n
a
to all
other alternatives in Time Interval j
()
im
Pa ε =
: the probability that Alternative
m
a
is uttered in Time Interval i
(| )
im j n
Pa a επ ==
: the probability that Alternative
m
a
is uttered in Time
Interval i given the condition that Alternative
n
a
is preferred the most in Time Interval
j
43
4.3 Preference Transition Model
This model relates the design team’s preference in the current time interval to that
in the next time interval. Individual designers may have different preferences, but in
this model, only the accumulative group preferences are considered in a probabilistic
relation.
In one time interval, it is assumed that there is an alternative which the team
prefers the most, called the most-preferred alternative, and the remaining alternatives
are the less-preferred alternatives. Each alternative has a probability to be the most-
preferred alternative and the less-preferred one. The probability of one alternative to
be most-preferred is the preferential probability of this alternative, and it describes the
likelihood that a team prefers this alternative over all others.
The Preference Transition Model is the mathematical implementation of
Assumption 4. At each transition, the design team can either 1) keep the most-
preferred alternative unchanged from the previous time interval or 2) change from the
most-preferred alternative to another alternative. The transition relationship between
one interval and the next depends on the preference strengths of the most-preferred
alternative and the less-preferred alternatives. In this study, a preliminary relationship
is approximated in which all less-preferred alternatives in the current time interval are
equally likely to become the most-preferred alternative in the next time interval.
The preferential probability for one alternative in the next time interval is
cumulative, and relates both the probability this alternative is most-preferred in two
44
consecutive intervals and the probability it transitions from less-preferred to most-
preferred. Two alternatives with different preference strengths have different
probabilities to be most-preferred and less-preferred, so the accumulated preferential
probabilities in the next time interval may differ even with this preliminary Preference
Transition Model.
In mathematical terms, the model can be expressed as in Equation (4.1).

1
(| )
1
1
ini m
pwhenn m
Pa a
p
when n m
N
ππ
+
= ⎧
⎪
⎪
⎪
== =
⎨
−
⎪
≠
⎪
⎪
⎩−
(4.1)

where
01 p ≤≤
is an hidden parameter, which means the probability that the most-
preferred alternative is kept unchanged from one time interval to the next consecutive
one. The bigger p is, the more consistent the preferences are over the design process;
and the smaller p is, the more frequently the preferences are changed.
4.4 Utterance-Preference Model
This model relates the team’s preference to the utterances of the alternatives in
the same time interval. In other words, it tries to approximate what designers think
with what designers say. This model is the mathematical implementation of
Assumption 5. In this model, a Vygotskian view [127] is adopted. This view differs
between two lines of speech, including inner speech and the external speech. The inner
45
speech reflects one’s meditation and can be regarded as a self-discussion process,
while external speech is used for social communications with others. Regarding the
study of relationship between psychological language and thought, Vygotysky [127]
said, “the area of inner speech is one of the most difficult to investigate”. In
engineering design, protocol analysis [25] is widely used to investigate the inner
speech for studying design activities and design decision making process. In this study,
both the inner speech from individual “think aloud” and the outer speech from team
conversation are considered. In the collection of verbal report data [40], it is assumed
that not all thoughts which pass through attention are verbalized and some thoughts
may be verbalized in variety of ways. Therefore, in the design selection process, the
concept regarding design alternatives may not be uttered in deterministic patterns.
Designers may utter the most-preferred alternatives in negative ways and the less-
preferred alternatives in positive ways due to the possible uncertainty in the process,
including the situations that designers do not know what they say, that designers do
not have deterministic preferences, that designers express their thoughts wrongly, and
that team dynamics influence the team work differently. Therefore, a probabilistic
model can be applied to explain this random relationship.
A design alternative can be uttered in a transcript in either a positive or negative
sense. When an utterance of an alternative has no negative words (e.g. “no,” “not,”
“hardly”) nearby in the transcript, this utterance is regarded as positive, otherwise it is
negative. The negative utterance of an alternative is counted as a positive utterance for
all the other alternatives. Another strategy would be to subtract the negative utterance
46
from the count of positive utterances, but this could lead to negative sums. Since the
model is probabilistic, it inherently considers the cases when an alternative is
mentioned that is not preferred the most. The model establishes the general pattern of
how often a design team mentions the alternative they prefer the most and how often
they mention the less-preferred alternatives. Similarly, a preliminary model is
approximated in which less-preferred alternatives are equally likely to be uttered by
designers in the same time interval.
The probability for one alternative is uttered is also a cumulative probability. It
relates both the probability this alternative is uttered when it is most-preferred and the
probability it is uttered while it is not most-preferred. For two alternatives with
different preference strengths, they have different probabilities to be most-preferred
and less-preferred, so the accumulated probabilities to be uttered may differ even with
this preliminary Utterance-Preference Model. In equation form:

(| )
1
1
in im
qwhenn m
Pa a
q
when n m
N
επ
= ⎧
⎪
⎪
⎪ ⎪
== =
⎨
−
⎪
≠
⎪
⎪ ⎪⎩−
(4.2)

where 01 q ≤≤ is an hidden parameter, which means the probability that the most-
preferred alternative is uttered in the discussion. In protocol studies of designers [25],
it is assumed that what designers say generally corresponds with what they think. In
47
this study, it is assumed that designers say what they prefer in most cases. i.e.,
1
1
q
q
N
−
>
−
. The reasonable value range for q is
1
1 q
N
< ≤ .
4.5 Preference Calculation
The preferences of the design alternatives in the design process may change over
time during the discussion of the design team. One of the objectives in this research is
to extract the preference evolution over the whole process. Although the preference
value of each alternative cannot be acquired from the method proposed in this research,
the probability of each design alternative to be most-preferred can be calculated.
The challenge: given the utterance data about the design alternatives and the
preferential probabilities of the design alternatives in the current time interval, what
are the preferential probabilities of the design alternatives in the next time interval?
By Law of total probability,

12 1
(|,,,...,)
ik ii i
Pa πσσσ σ
−−
=
=
11 1 1 1 2 1
1
( |,,...,, )( |,,...,)
i kii im imi i
mN
Pa aP a πσσ σπ π σσ σ
−− − −−
≤≤
===
∑ (4.3)

By Bayes Theorem,

111
( |,,...,, )
ik ii i m
Pa a πσσ σπ
−−
==
=
48
11 1 1 1 1
11 1 1 1 1
1
( | , ,..., , ) ( | ,..., , )
(| ,,...,, )( |,...,, )
ii K i i m i K i i m
ii n i i m i n i i m
nN
Pa aP a a
Pa aP a a
σπ σσπ π σσπ
σπ σσπ π σσπ
−− −−
−− − −
≤≤
=== =
=== =
∑
(4.4)

Equation (4.4) can be simplified into Equation (4.5) because the utterance data in
the current time interval are independent of the utterances in the historical time
intervals while given the preference in the current time interval, and the preference in
the current time interval is independent of the utterance data in the historical time
intervals while given the preference in the latest previous time interval.

11 1
( |,,...,, )
ik ii i m
Pa a πσσ σπ
−−
==
=
1
1
1
(| )( | )
(| )( | )
ii k i k i m
ii n i n i m
nN
PaP a a
PaP a a
σπ π π
σπ π π
−
−
≤≤
== =
== =
∑
(4.5)

Substituting Equation (4.5) back into Equation (4.3) gives the following two
equations ((4.6) and (4.7)).

When
2 i ≥
,
12 1
( | , , ,..., )
ik ii i
Pa πσσσ σ
−−
=
=
1
1121
1 1
1
(| )( | )
( | , ,..., )
(| )( | )
ii k i k i m
imi i
ii n i n i m mN
nN
PaP a a
Pa
PaP a a
σπ π π
πσσ σ
σπ π π
−
−−−
− ≤≤
≤≤
== =
=
== =
∑
∑
(4.6)

49
When 1 i = ,
11
(|)
k
Pa πσ =
=
11 1 0
0
11 0 1
1
(| )( | )
()
(| )( | )
kk m
m
ni n m mN
nN
PaP a a
Pa
PaP a a
σπ π π
π
σπ π π
≤≤
≤≤
== =
=
== =
∑
∑
(4.7)

Suppose the design alternatives are uttered
i
w times in the i
th
time interval, as
(1) (2) (3) ( )
, , ,...,
i
w
ii i i
aa a a , which are all in Alternative Set A. Assume that the utterances of
alternatives in one time interval strongly depend on the designers’ preference, and the
utterances of design alternatives are indifferent of each other given the strong
dependence on preferences, then

(| )
ii k
Pa σπ =
=
()
1
(| )
i
w
u
iik i
u
Pa a επ
=
==
∏
(4.8)

Equations (4.6), (4.7) and (4.8) recursively calculate the preference in the next
time interval from the preference in the current time interval. In order to make the
recursion work, two pieces of information should be given.
• The initial preferential probabilities of all alternatives before the first time
interval.
• The parameters of Preference Transition Model and Utterance-Preference
Model. i.e. Parameters p and q.
50

The first one can be resolved by several ways:
(1) Conducting surveys of designers before the start of the design process;
(2) Collecting preference information from an earlier design process;
(3) Analyzing preferences from the design of similar products;
(4) Establishing an unbiased starting point which assumes a uniform alternative
distribution.
In the case example in this study, because of the unknown initial preferences, all
alternatives are initiated with uniform alternative distribution, which gives an unbiased
starting point. i.e.,

0
1
( ) , 1,2,...,
k
Pa k N
N
π== =
(4.9)

Equation (4.9) means that before the discussion of the design process, all
alternatives have equal probability of being most preferred.
The parameters of two key models are not as easy to acquire as the initial
preferential probabilities of alternatives. In this research, an EM (Expectation-
Maximization) algorithm [28] is applied to searching the parameters of Preference
Transition Model and Utterance-Preference Model.
51
4.6 Estimation of Hidden Parameters
If the Preference Transition Model and the Utterance-Preference Model are given,
starting from the initial preferential probabilities at the beginning of the design
discussion, it is feasible to calculate the preferential probabilities for each design
alternative in each time interval and then to plot the preference evolution over the
whole design process. But the problem is that initially the parameters of these two
models are unknown. In this situation, utterance data are observable but preference
data are unobservable, and the models are incomplete because of the hidden
parameters. An EM (Expectation-Maximization) algorithm [28] is often used in
statistics for finding maximum likelihood estimates of parameters in probabilistic
models, where the model depends on unobserved hidden variables. In this study, it can
be applied to seek the values of the two hidden parameters of the two models.
An EM algorithm has two steps, the E-step and the M-step. The E-step estimates
the unobservable data. It can be accomplished by Equations (4.6) and (4.7). The M-
Step computes the maximum likelihood estimates of the parameters by maximizing
the expected likelihood found on the E-step. In this study, it corresponds to estimating
the values of p and q which make the utterance sample of the design alternatives occur
in the discussion with the maximal likelihood.
From Equation (4.1), it is known that
1
(| )
iki k
Pa a ππ
+
== is independent of k
and i. It means that no matter what time interval it is in, no matter which alternative
52
designers prefer the most in the previous time interval, designers have a fixed
probability to keep the most-preferred alternative unchanged.
By the maximum likelihood [43],
1
(| )
iki k
Pa a ππ
+
== can be estimated as

1
(| )
iki k
Pa a ππ
+
== =
1
111
1
111 1
(, )
(, )
imi m
iT m N
ini m
iT n N m N
Ca a
Ca a
ππ
ππ
+
≤≤ − ≤ ≤
+
≤≤ − ≤ ≤ ≤ ≤
==
==
∑∑
∑∑ ∑
(4.10)

where
1
(, )
ini m
Ca a ππ
+
== is a fractional count that counts the cases that
n
a is most-
preferred in the current time interval while
m
a is most-preferred in the previous time
interval.
1
(, )
ini m
Ca a ππ
+
== can be calculated as follows.

1
(, )
ini m
Ca a ππ
+
==
=
11 1 11
( | , ,..., ) ( | , ,..., )
ini i i mii
Pa P a πσσσπ σσ σ
++ −
==
(4.11)

The fractional counts are fractional numbers, and they are not the same as the
normal counting numbers, which are integers. But the values of fractional counts have
the proportional relationship with the integral numbers which count the cases when
1in
a π
+
= and
im
a π = , so fractional counts can be used in Equation (4.10) to estimate
of Parameter p.
53
Similarly, from Equation (4.2), (| )
ik i k
Pa a επ == is independent of i and k. It
means that in a certain time interval, designers have a fixed probability to utter the
same alternative as the one they prefer the most.
By the maximum likelihood, (| )
ik i k
Pa a επ == can be estimated as

(| )
ik i k
Pa a επ == =
11
11 1
(, )
(, )
imi m
iT m N
ini m
iT n N m N
Ca a
Ca a
επ
επ
≤≤ ≤ ≤
≤≤ ≤ ≤ ≤ ≤
==
==
∑∑
∑∑ ∑
(4.12)

where (, )
ini m
Ca a επ == is also a fractional count, which counts the number of cases
that
n
a is uttered while
m
a is most-preferred in the same time interval. It can be
calculated as follows.

11
(, ) ( )( |, ,...,)
inim in i m ii
Ca a C aP a επ ε π σσ σ
−
== = = = (4.13)

where ()
in
Ca ε = is the number of utterances of Alternative
n
a in the time interval i.
Equations (4.12) and (4.13) calculate q based on the samples of alternative
utterances and preferences. When using the above procedure to calculate q, it should
be noted that the value of q should be more than 1/N.
Because the EM algorithm is guaranteed to improve the probability of the sample
of alternative occurrences at each iteration, p and q will converge to values which try
54
to maximize this probability [13]. These converged values can be regarded as the
parameters for Preference Transition Model and Utterance-Preference Model. The
shortcoming of EM algorithm is that it may converge to a local optimum. Multiple
initial estimates can be used to avoid being trapped in a local optimum. Simulated
annealing can be combine with EM algorithm to overcome the local optima problem
[121, 122].
4.7 Time Intervals
In this research, it is assumed that designers do not change their preferences on
design alternatives in one time interval. Preferences can only be changed at the
transitions between time intervals. Based on the designers’ transcribed discussion,
there are several ways to determine time intervals.
1. Collect all transitional words (e.g., “but”, “however,” “while”) in the transcript,
and divide the transcript into varying time intervals with these transition words.
2. Collect all key design alternative occurrences from the transcript of designers’
discussion, and time-stamp all collected words. Big time gaps between the key
alternative words mark the separations of time intervals.
3. Set up a fixed word frequency. The process is divided into time intervals in
which there are equal numbers of word utterances of key alternatives.
4. Make a fixed time interval. The length of each interval depends on the desired
granularity of preference evolution.
55
Although Methods 3 and 4 are not as accurate as Methods 1 and 2, they are more
direct to implement. In this research, Method 4 is chosen and modified to specify the
time intervals in the design process. The time intervals are nearly of the same lengths
but not exactly equal because the divisions occurred only after one finished his/her
conversations and there was no immediate following-ups. If another designer was
ready to talk while one was still talking, divisions of time intervals would wait until
both finished. Even in this way, the real preference of the team may change inside the
interval as well. For the sake of simplicity, it is assumed that designers do not change
their preferences for design alternatives within a time interval in this study. The
preferences are considered accumulatively for each interval and the preference
changes are only considered between the intervals. The precise granularity of the
changes inside the time interval could be studied in future research.
4.8 Case Study
4.8.1 Case Background
This case example is the same as the one used for the thematic analysis of the
discussion transcript in Section 3.2. In that project, designers attempted to solve two
component selection problems for the large-scale space system re-design, with three
alternative candidates respectively in a time range of 3 sessions. The case study in this
section is focused on the first session of the second component selection problem.
56
4.8.2 Data Collection and Method Implementation
The transcript in the first session was input as the raw data in this case study. The
utterances of the three alternatives (represented by a
1
, a
2
, and a
3
) for the second
selection problem were collected in intervals of 10 minutes, as shown in Table 4.1.

Table 4.1 Sample Data: Utterances of Alternatives
Alternative

Interval
a
1
a
2
a
3

1 9 0 3
2 9 2 7
3 1 0 0
4 4 0 9
5 0 0 6
6 3 1 0
7 1 0 2
8 0 1 2
9 1 0 8
10 0 0 1
11 0 1 2
12 0 0 5

Initially, we can give any values to p and q if 0
is the scale factor of the Logit model. Let ε be the difference between
two ratings, the probability density function (pdf) and the cumulative distribution
function (cdf) of the logistic distribution on ε are shown in Equations (5.2) and (5.3).

2
()
(1 )
e
f
e
λε
λε
λ
ε
−
−
=
+
(5.2)

1
()
1
F
e
λε
ε
−
=
+
(5.3)

Figures 5.1 and 5.2 display the shape of the probability density function and
cumulative distribution function with different λ values. The cumulative distribution
function is also called sigmoid function because the shape of the probability function
resembles sigmoid curve.
70
0
1
2
3
4
5
6
-2 -1.5 -1 -0.5 0 0.5 1 1.5 2
Difference Between Two Utilities ( ε)
Probability Density Function
λ=1
λ=2
λ=5
λ=10
λ=20

Figure 5.1 Probability Density Function of Logistic Distribution

0
0.2
0.4
0.6
0.8
1
1.2
-2 -1.5 -1 -0.5 0 0.5 1 1.5 2
Difference Between Two Utilities ( ε)
Cumulative Distribution Function
λ=1
λ=2
λ=5
λ=10
λ=20

Figure 5.2 Cumulative Distribution Function of Logistic Distribution
71
In the study in this research, the ratings are in the range of [0, 1], so the possible
range for
ε
is [-1, 1], which differs from its original use with utilities with
[, ] ε ∈−∞ ∞ .
The positive scale parameter λ implies how large the tail of the logistical
distribution is. When it goes to infinity, all ratings are deterministic, and preferential
probabilities are either 0 or 1. When it becomes zero, all choices are equally likely to
be selected as the most-preferred. Therefore, the values of the preferential probabilities
translated from ratings with the Logit model highly depend on
λ
. This study considers
a possible range for the scale factor
λ
for comparison with PPT. For this study, the
following two questions must be answered for finding a range for
λ
.
Question 1: Given that the preferential probability of one alternative over another
is 0.5 when the difference between two ratings is 0, what is a reasonable preferential
probability if the difference of the ratings for those two alternatives is very small but
greater than 0? For example: suppose the aggregated rating for one alternative is 0.50,
and 0.49 for another. What is the probability that the team actually prefers the first
alternative over the second?
Question 2: Given that the preferential probability for one alternative over
another is 1 when the difference between two ratings is of the maximal possible
difference (1 in this study), what is a reasonable preferential probability if the
difference is very big but less than 1? For example: suppose the aggregated rating for
72
one alternative is 0.95, and 0.05 for another. What is the probability that the team
actually prefer the first alternative over the second?
Answers to these two questions can be used to estimate a reasonable range for λ ,
which can help determine the possible range for the converted preferential
probabilities.
5.3 Similarity Measures of Preferential Probabilities from Different Sources
This section provides three kinds of measures (geometric distances, cosine
similarities, and correlation coefficients) to compare the group preferential
probabilities translated from the surveys with the preferential probabilities extracted
from the design discussion transcript. The collection of preferential probabilities can
be represented in vector form, with each element representing a specific preferential
probability for an alternative at a specified interval. The differences and the
similarities between two vectors of preferential probabilities can be studied with the
following measures.
5.3.1 Geometric Distance
The distance indicates how far the two groups of preferential probabilities are
from one another. Assume there are two vectors of preferential probabilities, V and W,
with n elements in each vector.

12
(, ,..., )
n
Vvv v =

73
12
( , ,..., )
n
Www w =

where n is the total number of preferential probabilities attained either by PPT or PPS,
the product of the number of intervals and the number of alternatives.
(1 )
i
vin ≤≤
is
the i
th
preferential probability extracted from the transcript with PPT, (1 )
i
win ≤≤ is
the i
th
preferential probability transferred from the rating values on surveys.
There are several ways to measure distances derived from coordinate geometry,
including
1
L norm (Manhattan distance),
2
L norm (Euclidean distance),
p
L norm,
and L
∞
norm (Chebyshev distance). In research,
p
L norm is rarely used for the values
of p other than 1, 2 and ∞ . In a two-dimensional space,
1
L norm distance is like is
the distance a car would drive in a city laid out in square blocks (if there are no one-
way streets),
2
L norm distance is the length of the line between the two points, and L
∞

is the greatest distance along one of the coordinates. In this study, the samples of the
preferential probabilities in vectors and
1
L
,
2
L
and L
∞
norms are employed to measure
the distances of two groups of preferential probabilities.

1
1
(, ) | | | |
n
ii
i
LVW V W v w
=
= − = −
∑
(5.4)

74
2
2
1
(, ) || || ( )
n
ii
i
LVW V W v w
=
= − = −
∑
(5.5)

1/
1
(, ) || ||
p n
p
ppii
i
LVW V W v w
=
⎛⎞
⎟ ⎜
= − = − ⎟
⎜
⎟
⎟ ⎜ ⎜
⎝⎠
∑ (5.6)

1/
1
(, ) || || lim max( )
p n
p
ii i i
pi
i
LVW V W v w v w
∞∞
→∞
=
⎛⎞
⎟ ⎜
= − = − = − ⎟
⎜
⎟
⎟ ⎜
⎝⎠
∑ (5.7)

The smaller the distance, the smaller the average difference between these two
groups of preferential probabilities is. The norm distance descriptively imply how far
or close the two vectors are different from each other.
Figure 5.3 shows the distances in a two-dimensional space. Suppose the unit
length of the grid in Figure 5.3 is 1, the Manhattan distance (
1
L norm) between the
two black points is 12, as shown in red, blue or yellow lines, and the Euclidean
distance (
2
L norm) between those two points is
6 2 8.48 ≈
, as shown with the green
lines, and the Chebyshev distance (L
∞
norm) is 6, which is the maximal distance
along one dimension (x-axis or y-axis).

75

Figure 5.3
2
Manhattan Distance, Euclidean Distance and Chebyshev Distance

5.3.2 Cosine Similarity (Dissimilarity)
Cosine similarity depicts the angle between two vectors of preferential
probabilities. The larger the similarity value is, the closer the two sets of preferential
probabilities are.

1
22
11
cos( , )
|| || || ||
n
ii
i
nn
ii
ii
vw
VW
VW
VW
vw
=
==
==
∑
∑∑
i
i
(5.8)

2
Figure courtesy of Wikipedia, http://en.wikipedia.org/wiki/Taxicab_geometry
76
Values range from +1 to -1. When the cosine similarity is 0, it means that the two
preferences are geometrically orthogonal, and that the two groups of preferences are
totally unrelated. When it is +1, it means that they are perfectly positively correlated,
and when it is -1, it means that they are perfectly negatively correlated. In this study,
since all probability values are non-negative, the possible range for the cosine
similarity is [0, 1]. Figure 5.4 shows an instance of an angle between two vectors ((x
1
,
y
1
) and (x
2
, y
2
)) in a two-dimensional space. The cosine similarity value is the cos( θ)
where θ is the angle shown in Figure 5.4.

X
Y
θ
(x
1
, y
1
)
(x
2
, y
2
)

Figure 5.4 Cosine Similarity between Two Vectors in 2-D Space

5.3.3 Pearson Product-moment Correlation Coefficient
In statistics, the Pearson product-moment correlation coefficient is a common
measure of the correlation between two variables v and w. In this study, it is used to
77
measure the consistency of the preferential probabilities gained in PPT and PPS. The
preferential probabilities extracted from the transcript (PPT) and from the surveys
(PPS) are paired for analysis. Since the Pearson product-moment correlation
coefficient is measured in a sample, “Pearson’s r” is designated. Let v be the
preferential probabilities extracted from the transcript, and w be the preferential
probabilities transferred from the rating values given in the surveys.

1
()( )
(, )
(1)
n
ii
i
vw
vvw w
rvw
nss
=
−−
=
−
∑
(5.9)

where v and w are the sample means of v and w,
v
s and
w
s are the sample standard
deviations of v and w, and n is the sample size.
Values range from +1 to -1. A correlation of +1/-1 means that there is a perfect
positive/negative linear relationship between variables. The closer the Pearson
correlation coefficient is to the +1/-1, the better two variables are correlated
positively/negatively.
5.3.4 Spearman Ranks Correlation Coefficient
Spearman's rank correlation coefficient (Spearman's rho) is a non-parametric
measure of correlation – that is, it assesses how well an arbitrary monotonic function
could describe the relationship between two variables, without making any
78
assumptions about the frequency distribution of the variables. In principle, spearman's
rho ( ρ) is a special case of the Pearson product-moment coefficient in which the data
values in two data sets are converted to rankings before calculating the coefficient [89].
In practice, however, a simpler procedure is normally used to calculate ρ. If there is no
tied ranks, ρ can be given by Equation (5.10).

2
2
6
(, ) 1
(1)
i
d
vw
nn
ρ = −
−
∑
(5.10)

Where
i
d is the difference between the ranks of corresponding values
i
v and
i
w .
If there are tied ranks, Pearson’s product-moment coefficient between ranks is
used to calculate ρ, as shown in Equation (5.11).

11 1
22 2 2
11 1 1
()( )( )
(, )
()( ) ( )( )
nn n
ii i i
ii i
nn n n
ii i i
ii i i
nvw v w
vw
nv v n w w
ρ
===
== = =
−
=
−−
∑∑∑
∑∑ ∑ ∑
(5.11)

Spearman ranks correlation coefficients also range from +1 to -1. A correlation of
+1/-1 means that there is a perfect positive/negative linear relationship between
variables. The closer the Spearman ranks correlation coefficient is to the +1/-1, the
better two variables are correlated positively/negatively.
79
5.4 Experimental Study
In the large scale system design case example (Chapter 3 and 4), the team
designer’s intents of preference are not accessible, so it is impossible to study the
similarity and correlation between the transcript-extracted preferential probabilities
and the team’s real intent. In this chapter, a new experiment is conducted for
comparative study. It is a case study of a three-person design team working on two
design selection problems for coffee maker re-design. Individual design team
members were periodically asked to complete surveys of their preferences.
Preferential probabilities were then extracted from team discussion transcripts and
compared with the explicit preferences stated in surveys. An evaluating comparison is
conducted between the preference results of those two methods (Referring to Figure
1.1).
Table 5.1 summarizes the differences between the new experiment and the case
example used in the preliminary studies.

Table 5.1 Comparison of the New Experiment with the First Example
# People Level of
knowledge
Experiment
Length
Problem
Scope
Questionnaire
Experiment 1
(Space System
Re-design)
17 Experts 9 hours Large
Scale
N/A
Experiment 2
(Coffee Maker
Re-design)
3 Novices 1 hour Small
Scale
Every 10
minutes

80
5.4.1 Team Description and Analysis
Team Basic Information: The design team in the experiment was a group of 3
Industrial and Systems Engineering graduate students at the University of Southern
California. They are all currently enrolled in Ph.D. programs in Industrial and Systems
Engineering. One had background in Mechanical Engineering and 7 years of work
experience, one had a background in Electrical Engineering and 2 years of work
experience, and the third one had background in Mechanical Engineering without
work experience. They had known each other for about 2 years but they had never
worked together on a design project before. The demographic information for the three
designers (coded as D
1
, D
2
, D
3
) in coffee maker re-design is listed in Table 5.2.
Team Dynamics: In this experiment, all the designers are the author’s friends and
volunteer to participate in the experiment. The friendship and the award for
recognition are the main motivations for the team members. After reviewing and
analyzing the video of their teamwork after the experiment, it is found that although it
was the first time for them to work together, they had clear team goals, and they had
good trusts on the assigned team task. They hold different personalities. Their Mayer-
Briggs Typology Test [61, 68, 133] on the designers were ENFJ (Extraverted,
iNtuitive, Feeling, Judging), INTP (Introverted, iNtuitive, Thinking, Perceiving), and
ENTJ (Extraverted, iNtuitive, Thinking, Judging) respectively. During the discussion,
there existed conflicts, but the conflicts were mainly task conflicts instead of
relationship conflicts [116]. The conflicts in the team were drivers for the continuous
81
discussion and incurred the changes of the team preference changes. There was no
hierarchy in the team, and they took leadership shift to manage the team. During the
interviews after the design process, they all felt that they contributed equally in the
design process, and they had good collaborations on the team design. This was
confirmed by both reviewing the transcript and the recorded video.

Table 5.2 Demographic Information of the Team Members in Coffee Maker Re-
design Project
Designer D
1
D
2
D
3

Gender Male Female Male
Degree
Objective
Ph.D. Ph.D. Ph.D.
Major Industrial &
Systems
Engineering
Industrial &
Systems
Engineering
Industrial &
Systems
Engineering
Undergraduate
Major
Mechanical
Engineering
Electrical
Engineering
Mechanical
Engineering
Work
Experience
7 Years 2 Years No
Design
Experience
Novice Novice Novice
Mayer-Briggs
Typology Test
ENFJ (Extraverted,
iNtuitive, Feeling,
Judging)
INTP (Introverted,
iNtuitive, Thinking,
Perceiving)
ENTJ (Extraverted,
iNtuitive, Thinking,
Judging)
Design
Experience
Novice Novice Novice

5.4.2 Experiment Description
The team’s task was to decide on two component selection design problems, each
of which had three candidate alternatives that were provided a priori:
82
Imagine you are a retired person who is a coffee connoisseur. Your day cannot
begin until you make coffee each morning for you and your spouse. You are in good
health but are not as strong or mobile as you were when you were younger. As a
connoisseur, you prefer fresh ground coffee to instant coffee like Folger’s, and you are
well informed about the various types of gourmet coffee available, as well as the tools
and equipment to prepare it. However, you are now on a fixed income and are
conscious about how you spend your money which is why you make coffee at home
rather than visit Peet’s every morning.
One of the issues in studying preferences is that they may be ambiguous. Hey,
Kulok and Lewis [57, 75] note that human designers may not be consistent when they
state their preferences explicitly. This has the potential to make quantitative analysis
of surveyed preferences difficult. The tack taken in this paper is to examine overall
trends in preferences across a number of design alternatives, rather than assume that
the findings for one alternative at one point in time are correct.
The objective of the experiment is to select two components (carafe and filter) for
the coffee maker. Although coffee maker has more than two components and the
design of coffee maker is much more complicated, a two-component selection
problem is appropriate for a novice design team and is helpful for a preliminary
research of extracting preference information from transcripts. The total cost for these
two components together cannot exceed $35.00. Table 5.3 lists the three carafe
alternatives (glass, stainless-steel, and plastic). Table 5.4 lists the three filter
83
alternatives (gold tone, paper, and titanium). The designers are provided additional
features and specifications (e.g. cost, fragility, heat retention, and etc.) that may play a
role in their preferences for the carafe and filter in Table 5.3 and Table 5.4.
In the process of design selection, they are encouraged to communicate only by
verbal discussion, and when they use any charts, graphs, or tables, they need “think
aloud” the information that they want convey and the listeners need “think aloud” [25,
41] the information they get in their mind.
Before the experiment, each team member was given a think-aloud training
exercise so that they would learn to name each alternative with proper names rather
than ambiguous pronouns (“this” or “that”), in order to facilitate the tracking of design
alternatives in the transcript. For example, in the carafe selection, glass carafe can only
be called as “glass carafe”, “glass pot”, “glass coffee carafe”, “glass coffee pot”,
“carafe A”, “pot A”, or “glass alternative”. During the experiment, they discussed
their preferences and rationale with each other until a consensus was reached. This
discussion was recorded and transcribed for Approach PPT. During the same exercise,
they were asked to fill out surveys expressing their preference ratings for design
choices. The surveys used in the experiment are attached in Appendix A.

84
Table 5.3 Design Information for Carafe Selection
Name/ID Glass coffee pot
Glass coffee carafe
Coffee pot A
Carafe A
Steel coffee pot
Steel coffee carafe
Stainless-steel
carafe
Coffee pot B
Carafe B
Plastic coffee pot
Plastic coffee
carafe
Coffee pot C
Carafe C
Photo

Description Glass with warming
plate
thermal-insulated
stainless-steel
thermal-insulated
plastics (inside
glass)
Cost $10.00 $20.00 $15.00
Warming plate
cost
$5.00 0 0
Footprint size Big Small Small
Fragility Fragile Strong Fragile material
inside
Durability
(reliability)
Durable Durable Less durable
Heat retention Good with heating
plate
OK with double
layers of steel
Good with mirror
glass inside
Weight Light Heavy Light
Portability Not portable Portable Portable
Easy to clean Easy to clean Not easy to clean Not easy to clean
Style and
aesthetic value
Moderate attractive Very attractive Not attractive
Capacity Can be designed as
wanted
Available for 2
cups and 6 cups
Can be designed as
wanted
Available for 2
cups and 6 cups
Can be designed as
wanted
Available for 2
cups and 6 cups
Spout Not dribbles after
pouring
Dribbles after
pouring
Dribbles after
pouring
Can tell how
much coffee is
left
Yes No No

85
Table 5.4 Design Information for Filter Selection
Name/ID Gold tone filter
Filter A
Paper filter
Disposable filter
Filter B
Titanium filter
Ti filter
Filter C
Photo

Description Permanent Gold
Tone filter
disposable paper
filter
Ti-Titanium
Permanent Coffee
Filter
Cost $9.99 $3.99/100 $19.99
Durability Permanent use Disposable Permanent use
Styling Neutral golden
color
N/A Fashionable color
Cleanability Clean after use, and
will never stain
Disposable
Single use
Clean after use,
may stain
Portability Not foldable Collapsible/foldabl
e, easily portable
Not foldable
Easy to remove
from the cone
No No Yes, with the
handle
Coffee aroma
retention
Can absorb and
retain coffee aroma
N/A Good at absorbing
and retaining coffee
aroma with
continuous coffee
cooking

Studies of team discussion suggest that team members enter into discussion
armed with only partial, independent knowledge of the topic at hand and part of the
goal of group discussion is to elicit this partial knowledge so that better decisions may
be made [46]. In order to encourage discussion among the participants and to better
simulate a real-world team experience, information about the design choices was
provided in the following ways:
86
1. Each individual was armed with different perspectives of detailed
information regarding design alternatives. In this way, the team members
would realistically discuss features of the alternatives with each other in
order to uncover information about the other alternatives.
2. Surveys were conducted individually, and no one overhears the ratings given
by the others.
3. Each designer was encouraged to give a brief rationale for their rating and
ranking to decrease the possibility that they gave arbitrary ratings.
The duration of the experiment was 50 minutes, including 10 minutes for
instructions and training, 8 minutes for filling out 5 surveys during the design process,
and 4 group discussion intervals that lasted about 8 minutes each.

5.4.3 Transcript Analysis
The entire discussion of the team was audio- and video- recorded and transcribed.
A segment sample of the discussion transcript is shown in Appendix B. Approach PPT
was applied to the transcripts. The utterances of the six alternatives (three alternatives
each for component selection problem) were collected in intervals of 8 minutes to
match the intervals at which the questionnaires were administered, as shown in Table
5.5. Designers had discussion on Carafe Selection Problem all though the design
process, while didn’t start discussing Filter Selection Problem until Time Interval 2.
87

Table 5.5 Sample Data: Utterances of Alternatives
Carafe Filter Alternative

Interval
Glass Steel PlasticGold tone Paper Titanium
1 13 8 7 0 0 0
2 12 7 6 3 5 2
3 11 6 1 8 15 10
4 9 3 0 5 6 14

While applying the Preference Transition Model and the Utterance-Preference
Model on the utterance data, we can give any initial values to p and q if 0(l+u)/2,
1
λ
will be positive, and the distribution becomes a mirrored truncated exponential
distribution decaying from the upper bound to the lower bound. There are three more
extreme cases: (1) when r=(l+u)/2,
1
λ will become 0, and the distribution is reduced
to a uniform distribution between the lower and upper bounds; (2) when r=l,
1
λ will
be negative infinity, and the distribution is reduced to a Dirac delta distribution at the
lower bound, which means the alternative is not accepted; (3) when r=u,
1
λ will be
positive infinity, and the distribution is reduced to a Dirac delta distribution at the
upper bound, which means the alternative is accepted for sure. Figures 1-5 show the
the distribution instances under the principle of maximum entropy for the stated rating
0, 0.2, 0.5, 0.8, and 1 when the bounded range is [0, 1].

0
1
2
3
4
5
0 0.2 0.4 0.6 0.8 1
Rating
Probability Distribution Function

Figure 6.1 Rating Distribution with Stated Rating=0

111
0
1
2
3
4
5
0 0.2 0.4 0.6 0.8 1
Rating
Probability Distribution Function

Figure 6.2 Rating Distribution with Stated Rating=0.2

0
1
2
3
4
5
0 0.2 0.4 0.6 0.8 1
Rating
Probability Distribution Function

Figure 6.3 Rating Distribution with Stated Rating=0.5

0
1
2
3
4
5
0 0.2 0.4 0.6 0.8 1
Rating
Probability Distribution Function

Figure 6.4 Rating Distribution with Stated Rating=0.8
112
0
1
2
3
4
5
0 0.2 0.4 0.6 0.8 1
Rating
Probability Distribution Function

Figure 6.5 Rating Distribution with Stated Rating=1

However, in some survey cases, the ratings are given in a relative way rather than
an absolute way. For example, the sampling from the distribution function may have
an implicit constraint that the sampling ratings from the probability distribution should
sum up to a certain value, such as when designers have 10 points to allocate among the
alternatives.
Suppose an individual designer is given W points among N alternatives. The
relative ratings the designer assigns are r
1
, r
2
, … r
N
. The possible relative values for
Alternative i (1 iN ≤≤ ) are in the range [l
i
, u
i
]. Let x
1
, x
2
, … x
N
be the sampling
variable for the relative ratings for each alternative. The joint distribution function can
be represented by

f(x
1
, x
2
, … x
N
)

113
With the constraint that x
1
+x
2
+ …+x
N
=W, the function can be reduced to

f(x
1
, x
2
, … x
N-1
)

Similarly, by maximizing the entropy of the joint distribution function with the
constraint that the expected value on variable x
i
is r
i
, f(x
1
, x
2
, … x
N-1
) can be
represented as in the Equation (6.7).

12 1
( , ..., )
N
fxx x
−
=

1
1
i
11 2 2 11
1, 1

[]
otherwise
...
0

0
N
Nk N
k
ii
NN
N
if W u x W l
and l x u i
xx x
e
λλ λ
λ
+
=
−−
−
−≤ ≤ −
≤≤ ∀ ∈
+++
⎧
⎪
⎪
⎪
⎪
⎪
⎪
⎪
⎪ ⎪
⎨
⎪
⎪
⎪
⎪
⎪
⎪
⎪
⎪
⎪ ⎩
∑
(6.7)

The above distribution function shows that the joint exponential distribution is
only meaningful when all the sample variables are in the possible ranges, otherwise it
is zero.
01 1
, ,...,
N
λλ λ
−
can be solved from the following N equations.

114
12 1
12 1
12 1
... ( , ..., ) 1
N
N
bb b
N
aa a
fx x x
−
−
−
=
∫∫ ∫

12 1
12 1
11 2 1 1
... ( , ..., )
N
N
bb b
N
aa a
xfx x x r
−
−
−
=
∫∫ ∫

12 1
12 1
21 2 1 2
... ( , ..., )
N
N
bb b
N
aa a
xfx x x r
−
−
−
=
∫∫ ∫

┋
12 1
12 1
11 2 1 1
... ( , ..., )
N
N
bb b
NN N
aa a
xfxx x r
−
−
−− −
=
∫∫ ∫

The first equation guarantees that total probability is 1 integrated over the
possible rating ranges for the joint distribution, and the next N-1 equations set the
requirements for the expected values for variable x
1
to x
N-1
. The expected value for
variable x
N
is met tacitly because E(x
N
)=E(W-x
1
-x
2
-…x
N-1
)= W-r
1
-r
2
-…r
N-1
= r
N
.
Figure 6.6 shows the rating distribution for 3 alternatives with average 0.1, 0.2
and 0.7 in the range [0, 1]. There is no constraint on the sum of the samples from the
distributions. While Figure 6.7 shows the case when a constraint holds that the sum of
the three sampled ratings from these three distributions must be 1. In comparing
Figure 6.7 with Figure 6.6, it is observed that in Figure 6.7, the distribution drops
down near the upper bound, and is especially obvious for distributions with high
expected values. In this case, the sampled rating for one alternative is determined by
the sampled ratings for all other alternatives. If the sum of the ratings for all other
alternatives is greater than 1, then the set of samples is invalid and has to be
115
disregarded because it does not meet the constraint. With this constraint, points with
higher values in the distribution are more likely to be dropped.

0
2
4
6
8
10
0 0.2 0.4 0.6 0.8 1
Rating
Probability Distribution Function
Expected Rating=0.1
Expected Rating=0.2
Expected Rating=0.7

Figure 6.6 Instances of Rating Distribution without Sum Constraint

0
2
4
6
8
10
0 0.2 0.4 0.6 0.8 1
Rating
Probability Distribution Function
Expected Rating=0.1
Expected Rating=0.2
Expected Rating=0.7

Figure 6.7 Instances of Rating Distribution (Sum of Three Sampled Ratings = 1)
116
6.2 Construction of Probabilistic Distribution for Group Preferences
Understanding a team’s preferences is useful for design, and often the case that
members of a team have different preferences. Determining a team rating from
individual ratings is challenging for a number of reasons, and two are considered here.
One is that any individual rating given can be uncertain, either due to fuzziness (does
the person think of a rating of 0.3 or 0.4 as roughly the same?) or simple human error.
The second challenge is the role of team organizational issues and social dynamics.
There may be differences in opinion among individual designers which may lead to
different group ratings. There is a rich literature on decision-making styles in groups
and how the opinions of team members and team leaders may be aggregated [126],
such as Pairwise Comparison Charts (PCC) [39], Axiomatization with Cardinal Utility
[69] and Distance-Based Collective Preorder Integration [63].
In this research, it is assumed that group ratings are bounded somewhere between
the highest individual rating and the lowest individual rating. The construction of the
distribution for team ratings is similar to the approach described in Section 6.1, and
includes information about the lower bound, the upper bound and the weighted
average rating. The weighted average rating can be estimated in several ways. If there
is no hierarchy and no apparent leader in the team, the weightings can be simply
assumed equal or proportional to an individual’s utterances in the discussion.
Weightings may be adjusted to reflect information such as an individual’s leadership,
expertise, and member importance [62, 107].
117
6.3 Simulation of Group’s Preferences
Monte Carlo simulation was used to determine the chances that one alternative
has a higher group rating than any other alternative. In each round of simulation,
individual ratings are sampled from the probabilistic distribution constructed in
Section 6.1, and then group ratings for each alternative are found from the
probabilistic distribution in Section 6.2. By comparing the group ratings of the
alternatives, the most preferred alternative is determined. Repeating this process in
simulation, the probability that one alternative is preferred over all other alternatives
can be estimated from statistical results from simulation. The preferential probability
for an alternative approximately equals to the proportion of rounds when this
alternative has the highest group rating in the simulation.
When there is no constraint on the sum of the sampled ratings on all alternatives,
the steps of the simulation process are as follows:
1. Construct a distribution for each individual member’s rating for each
alternative, as described in Section 6.1;
2. From each distribution, randomly select a sample as the “true” individual
rating;
3. For each alternative, based on the sampled individual ratings, construct a
distribution for the possible group rating, as described in Section 6.2;
4. Sample to get the group rating for each alternative;
118
5. Compare the group ratings to determine the most-preferred alternative;
6. Repeat Step 2-5 until the predefined maximum number of simulation runs is
reached. The group’s preferential probabilities can be estimated from the
obtained statistical simulation results.
If there is a constraint that the total sampled ratings on all alternatives are fixed
(e.g. designers allot a fixed number of points to the alternatives), then the joint
distributions for both individual and group ratings are used as described in Section 6.1
and 6.2, and only (N-1) alternative ratings are to be sampled, the left one is determined
by subtracting the sampled ratings from the predetermined sum, e.g., 1 when the
ratings are normalized values as shown in this study.
In the simulation for this study, the rejection method [98, 103] can be employed
to generate the samples for the distribution functions. The rejection method can
generate sampling values from an arbitrary probability distribution function.
6.4 Experimental Study of PPT and PPS
6.4.1 Case Background
The exemplified case for comparing PPT and PPS in this chapter is also the one
used in the previous chapter. The design team is a team of 3 engineering graduate
students. The team’s task was to select a carafe component and a filter component for
a coffee maker re-design project from a set of alternatives. This discussion was
119
recorded and transcribed for PPT. During the same exercise, they were asked to fill
out surveys expressing their preferences for design choices.
6.4.2 Survey Results with PPS
Five surveys were administered all along the design process, including one before
the discussion started (Time=10:00), one after the discussion ended (Time=48.20), and
three within the discussion (Time=20:06, 30:12, 40:00). The times are expressed as
mm:ss. The rating results are shown in Section 5.4.4 (Tables 5.8-5.12).
PPS was applied to convert these ratings into preferential probabilities. Since the
ratings in the experiment were normalized relative preference ratings, there was a
constraint on the fixed sum. The joint distributions descried in Sections 6.1 and 6.2
were used to for sampling the individual ratings and the group ratings for simulation.
For example, in this case study with three alternatives, for a certain designer, if the
sampled ratings for the first and the second alternative were 0.2 and 0.5, then the
sampled rating for the third one would be 1-0.2-0.5=0.3 by default. And the sampled
results would be dropped when the sum of the first two ratings was greater than 1
because it conflicted with the constraint.
The sampled individual ratings are imported for constructing the group rating
distribution, and then a group rating is sampled from the distribution. The weighted
average is one of the constraints for solving the parameters for the distribution. In this
experiment, the designers were interviewed after the design about the contributed
work, and the videotape was reviewed again for team dynamics analysis. It is noticed
120
that all team members contributed almost equally, so equal weightings on the
individual survey analysis were employed. The resulting values for each of the 5 time
intervals are shown in Table 6.1.

Table 6.1 Group Preferential Probabilities from Surveys (PPS)
Carafe Filter Alternative

Interval
Glass Steel Plastic Gold
tone
Paper Titanium
0 0.492 0.482 0.0259 0.131 0.719 0.150
1 0.654 0.322 0.0232 0.161 0.815 0.0240
2 0.680 0.272 0.0482 0.0977 0.885 0.0172
3 0.884 0.101 0.0149 0.0319 0.929 0.0390
4 0.885 0.101 0.0148 0.101 0.0693 0.830

The survey before the design process (Interval 0) shows that both the glass carafe
and the steel carafe have a ~49% chance to be selected as the “best” or most preferred
choice, while the plastic carafe has only a ~3% chance to be selected as the “best” or
most preferred choice. From the above data, it can be inferred that, as a group, the
glass and the stainless-steel carafes were preferred in the beginning, but that only the
glass one was preferred in the end. For the filter design, the design team preferred the
paper throughout the session until the very end when the Titanium filter became the
most preferred choice.

121
6.4.3 Transcript Analysis with Initialized Preferences
The entire discussion of the team was audio- and video- recorded and transcribed.
PPT was applied to the transcripts. The utterances of the six alternatives (three
alternatives each for the two component selection problems) were collected in
intervals of ~8 minutes (~10 minutes including survey filling) to match the intervals at
which the questionnaires were administered. The initial preferences at the beginning
of the design discussion can be given in several ways: 1) equally divided; 2)
preference information collected from an earlier design process; 3) analysis on
previous preference of similar designs; 4) conducting surveys before the design
process. In previous chapter, equal initial preferences were used. This time, the
preferential probabilities were initialized with the probability values translated from
the preference ratings on the survey which was done before the design process started.
Table 6.2 shows the results of preferential probabilities extracted from the discussion
transcripts. The ones in Interval 0 mean the initial preferential probabilities for starting
PPT.

Table 6.2 Group Preferential Probabilities from Transcripts (PPT)
Carafe Filter Alternative

Interval
Glass Steel Plastic Gold
tone
Paper Titanium
0 0.492 0.482 0.0259 0.131 0.719 0.150
1 0.856 0.137 0.00658 0.131 0.719 0.150
2 0.958 0.0395 0.00199 0.162 0.750 0.0872
3 0.988 0.0120 3.34E-05 0.00883 0.961 0.0300
4 0.997 0.00211 3.51E-05 0.00309 0.00927 0.988
122
Figure 6.8 and Figure 6.9 show the evolution of the preferential probabilities
according to the data in Table 6.2. Comparing Figure 6.8 and Figure 6.9 with Figure
5.5 and Figure 5.6, the preferential probabilities are different in the first few time
intervals (mainly first two time intervals). The reason is that the preferential
probabilities depend on both the utterance data in current interval and the preference
data in previous interval. Different initial preference data could greatly influence the
first few preferential probabilities in the application of Approach PPT. But the later
preferential probabilities are approximately identical, it means that even without the
initial preference information, Approach PPT is still a feasible way to predict the
preference information during the process.

0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
12:00 22:45 32:30 41:22 50:20
Time (mm:ss)
Preferential Probability
glass
steel
plastic

Figure 6.8 Design Process Evolution: Group Preferential Probabilities of the Three
Alternatives for Carafe Selection (Initial Probabilities with Pre-design Surveys)
123
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
12:00 22:45 32:30 41:22 50:20
Time (mm:ss)
Preferential Probability
gold
paper
Ti

Figure 6.9 Design Process Evolution: Group Preferential Probabilities of the Three
Alternatives for Filter Selection (Initial Probabilities with Pre-design Surveys)

6.5 Comparisons and Discussions
To better validate the probabilistic approach for extracting the preferential
probabilities from the transcript, a comparison between the preferences from the
transcript (PPT) and from surveys (PPS) was done in multiple ways: graphically, and
through geometric distance, cosine similarity, and correlation.
Figure 6.10 through Figure 6.12 overlay the evolutions of the preferential
probabilities from transcript analysis (PPT) and surveys (PPS) for the carafe selection.
They suggest that the glass carafe dominates over the other two alternatives (stainless-
steel carafe and plastic carafe) during the whole design process. Figure 6.13 through
Figure 6.15 show the results for filter selection. They indicate that the team’s
124
preferential probability is highest for the paper filter until the last interval, in which
discussion changed to the titanium filter.
The trends and the changes of the evolution are almost consistent in Figure 6.10
through Figure 6.15, which graphically indicate the transcript reflects the trends in the
designers’ preference in the case study. The conjecture from the charts is consistent
with the qualitative reading of the transcript. The team changed their choice on the
filter because further information was given in the design process that the glass carafe
and the paper filter could not function together, and so they had to select again. They
changed the filter option because they agreed that the filter was less important.

0
0.2
0.4
0.6
0.8
1
12:00 22:45 32:30 41:22 50:20
Time (mm:ss)
Preferential Probability
PPS
PPT

Figure 6.10 Comparison of Group Preferential Probabilities on Glass Carafe

125
0
0.2
0.4
0.6
0.8
1
12:00 22:45 32:30 41:22 50:20
Time (mm:ss)
Preferential Probability
PPS
PPT

Figure 6.11 Comparison of Group Preferential Probabilities on Stainless-steel Carafe

0
0.2
0.4
0.6
0.8
1
12:00 22:45 32:30 41:22 50:20
Time (mm:ss)
Preferential Probability
PPS
PPT

Figure 6.12 Comparison of Group Preferential Probabilities on Plastic Carafe

126
0
0.2
0.4
0.6
0.8
1
12:00 22:45 32:30 41:22 50:20
Time (mm:ss)
Preferential Probability
PPS
PPT

Figure 6.13 Comparison of Group Preferential Probabilities on Gold Tone Filter

0
0.2
0.4
0.6
0.8
1
12:00 22:45 32:30 41:22 50:20
Time (mm:ss)
Preferential Probability
PPS
PPT

Figure 6.14 Comparison of Group Preferential Probabilities on Paper Filter

127
0
0.2
0.4
0.6
0.8
1
12:00 22:45 32:30 41:22 50:20
Time (mm:ss)
Preferential Probability
PPS
PPT

Figure 6.15 Comparison of Group Preferential Probabilities on Titanium Filter

The three types of similarity measures proposed in Section 5.3 were applied on
the experimental results for quantitatively studying how the preferential probabilities
from PPT and PPS are distinctive or similar. All the data are collected from Table 6.1
and Table 6.2. The results of the L
1
norm, the L
2
norm, the cosine similarity, and two
correlation coefficients are shown in .
In the case study, there are 30 data points with totally 6 alternatives at 5 time
points in both data vector of PPT and PPS. Considering the initial values of PPT were
assigned according to the surveys results before the design started, the preferential
probabilities equal for both PPT and PPS at Time=12:00 (Interval 0), so the data in
Time Interval 0 are excluded for the calculation of the similarity measures. And since
the sum of the preferential probability values for three alternatives for each selection
problem is fixed, the data for one of the three alternatives are excluded as well. This
128
study uses the data from Glass Carafe, Stainless-Steel Carafe, Gold Tone Filter, and
Paper Filter for comparison. Therefore, in each vector of preferential probabilities,
there are 16 points for comparison.
In the comparative study in Chapter 5, only 14 points are applied for analysis,
while there are two more in this chapter. The reason is that in Chapter 5, the equal
likelihood is used to initialize the preferences before the design process for all the
alternatives. So those initial preferences are not correct. If the team does not discuss
the alternative in the first time interval, then the preferential probabilities predicted
from the initial preference are not correct as well. As the team only started discussing
the filters in Time Interval 2, the preferential probability values for filters in Time
Interval 1 should be dropped for accurate evaluation when using the uniform
distribution for initial preferences.
In this chapter, the preference information before the design process was taken
into account for applying Approach PPT. When the team does not talk about the filter
alternatives in Time Interval 1, the predicted preferential probabilities in Time Interval
1 is estimated to be the same as the ones before the design process. Since the initial
preferences are correctly given, the preferential probabilities in Time Interval are
correctly estimated as well, even without the utterance data.
Let
Sij
v be the preferential probabilities on the i
th
alternative of the component
selection Problem S (S=A: carafe, S=B: filter), extracted from the discussion transcript
(PPT) at the j
th
interval.,
Sij
w be the corresponding preferential probabilities translated
129
from the surveys (PPS). For example,
11 A
v is the PPT-extracted preferential
probabilities on Alternative A1 (glass carafe) at the first time interval.
Let V be the vector of all preferential probabilities extracted from the discussion
transcript (PPT); W be the vector of all preferential probabilities translated from the
surveys (PPS). This time, V and W are vectors of 16 elements each, with 4 selective
alternatives out of 6 at 4 time points when the results are compared. Table 6.3 shows
the possible ranges and the quantitative similarity measures between these two vectors,
with the formulas drawn from Section 5.3.

11 12 13 14
21 22 23 24
11 12 13 14
21 22 23 24
(, , , ,
, , , ,
, , , ,
, , , )
AA A A
AA A A
B BBB
BB B B
Vv v v v
v vvv
v vvv
vv v v
=

11 12 13 14
21 22 23 24
11 12 13 14
21 22 23 24
( , , , ,
, , , ,
, , , ,
, , , )
AA A A
AA A A
BB B B
B BBB
W w www
w www
w www
w www
=

There are 16 elements in each vector of V and W, and each element is in the
range of [0, 1]. Therefore, the possible L
1
metric distance between V and W is [0, 16],
and the possible L
1
metric distance between V and W is [0, 16 ], or [0, 4], with the
130
smaller value meaning the closer relationship of the two vectors. L
1
(V, W) = 1.84
means the difference between the results of PPT and PPS is 11.5% of the maximal
possible difference in L
1
norm space, and L
2
(V, W) = 0.543 means the difference is
13.5% of the maximal possible difference in L
2
norm space. Monte Carlo simulation
shows that the average distances and the standard deviations for two random feasible
sequences of 16 preferential probabilities are
1
L =4.27 (
1
L
s =0.864) and
2
L =1.31
(
2
L
s =0.238). Comparatively speaking, both two norms of distances between PPT and
PPS (L
1
= 1.84, L
2
= 0.543) are relatively close. It indicates the difference between the
vectors of preferential probabilities sourced from transcript and surveys is relatively
small.

Table 6.3 Experimental Results of Similarity Comparison between PPT and PPS
Measure Possible
Ranges
Results
L
1
norm [0, 16] 1.84
L
2
norm [0, 4] 0.543
Cosine similarity [0, +1] 0.974
Pearson product-moment
correlation coefficient
[-1, +1] 0.956
p-value: 7.67E-9
Spearman’s rho [-1, +1] 0.833
p-value: 6.56E-5

Cos(V, W) = 0.974 is very close to 1, implying that a high similarity is found
with the vectors of the preferential probabilities from transcript and from the surveys.
High correlation coefficients (Pearson’s and Spearman’s) are consistent with the
131
cosine similarity, which also shows a highly positive correlation. It shows that the
evolution of the transcript goes consistently with the evolution of the surveys in terms
of preferential probabilities in our experiment. P-value is the significant level for
testing the hypothesis of no correlation against the hypothesis with a non-zero
correlation. Small p-values confirm that there is a significant correlation between PPT
and PPS.
In total, the data of distances, cosine similarity, and Pearson correlation
coefficient all imply the consistency of the transcript and the survey in the experiment,
and point to the effectiveness of the probabilistic approach for extracting the
preferences from the transcript.
6.6 Remarks
This chapter assumes the preference ratings could be random for both individuals
and the team, and establishes a probabilistic approach (PPS) to translate preference
ratings into preferential probabilities under the principle of maximum entropy. As an
explicit counterpart of the implicit PPT, the preferential probabilities translated with
PPS can be applied to quantitatively validate the preferential probabilities extracted
with Approach PPT. The study in this chapter graphically and quantitatively evaluates
the preferential probabilities from transcripts with those from the surveys. The
consistent results of the case study further validate the effectiveness of the
probabilistic approach (PPT) proposed in Chapter 4. It is expected that the design
preferences may oscillate in the design process, and it is verified both in the surveys
132
and the transcripts in the experiment done for the case study in this chapter. The
probabilistic ways to describe how a design team prefers an alternative over the others
may lead to a novel way to understand the nature of a team’s preferences over time.
The probabilistic approach (PPS) preliminarily tries the links between the
traditional preference ratings and the preferential probabilities and may enlighten the
research on converting the preferential probabilities to the traditional preference
ratings. In this work, while applying PPS, the maximum entropy is employed to
construct the rating distributions for the rating preferences. Truncated exponential
distribution is used under the conservative way when a bounded rating value is given.
Since this method does not assume any distribution or parameter a prior, it is scalable
when additional information is provided.

133
Chapter 7 Conclusions and Future Research
7.1 Conclusions
This research examines discussion transcripts of an engineering design team, and
proposes a probabilistic approach to extract the team’s design preferential probabilities
from the transcripts. For evaluations, this research develops two approaches to
converts the preference ratings from the surveys taken at intervals in the design
process, and comparatively studies the possible consistency between the preferential
probabilities from two sources (the transcripts and the surveys). This research answers
the initial four questions posed in Introduction.
What design information can be gained from the design discussion transcripts?
Design team discussion includes the opinions of multiple team members, rather than
only individual contributions, as they discuss a design. The deliberation that goes on
within a team can have major influence on the choices that are made for a design. The
case studies in this research demonstrate that team discussion reflect the change of a
team’s preferences continuously over time, offering an opportunity to extract
preference-related and represent it as it evolves. In this research, the extracted design-
specific information reflects the evolution of the selection process. The collected
utterance information on design selection is evaluated to be a good source for
extracting the preference information in terms of preferential probabilities, which
implies how the entire team would like to select an alternative as “most-preferred”.
134
How can the preference information of a design team be computationally
extracted from their discussions? The method proposed in Chapter 4 links the
relations between what designers say in discussion and what they prefer in design with
an utterance-preference model and a preference transition model, and presents a
probabilistic approach to extracting preferential probabilities from transcripts of
design team discussion based on a time interval-based profile. It works to extract
preference information in an implicit way and as if the group is a single entity. The
probabilistic approach (PPT) for extracting the group’s preferential probabilities from
the transcript may be a potentially effective way to link the qualitative design
information that is generated by design teams with more quantitative design decision
making methods through the use of preference information.
How do the design preferences of a team evolve over time as the team changes its
priorities on the alternatives? This research shows how the design selection process
evolves over the design process. In the design selection process, when designers make
trade-offs, the design team’s preference may oscillate as well during the process.
From perspective of information entropy, a conjecture is that a design team starts with
lower certainty (higher entropy) and was more certain on the most-preferred choice
when the team reached the agreement. The results of this research on the two case
studies are consistent with this conjecture. One comment is that when additional
information is given to the team, this might not be true because the team may re-assess
all alternatives. The work in this research on time-based extraction of preferential
135
probabilities may lead to a novel way to understand the evolving nature of a team’s
preferences over the life of a project.
Is the preference information extracted in proposed method consistent with the
actual preferences? This research employs both qualitative reading of transcripts and
quantitative comparison with survey ratings to evaluate the preferential probabilities
extracted from the transcripts. Both results show the consistency for the case studies
and further validate the effectiveness of PPT. Two approaches including the Logit
Model (Chapter 5) and PPS (Chapter 6) are employed for graphical comparison and
quantitative comparison on geometric distances, cosine similarity and correlation
coefficients. The Logit Model is a direct and quick way to convert ratings to
preferential probabilities for comparison purposes, but because of its limitations on
design preferences, another approach (PPS) under the principle of maximum entropy
is established as a benchmark to quantitatively evaluate the probabilistic approach for
extracting the group’s preferences from the transcript. These two approaches
preliminarily try the links between the traditional preference ratings and the
preferential probabilities and may enlighten the research on converting the preferential
probabilities to the traditional preference ratings.
The approach proposed in this study for implicitly extracting the preference
information from the discussion transcript can be extended to other fields that include
a process of selecting among alternatives. It can be extended for use in economics to
extract consumers preferences from group discussion such as focus groups. However,
136
there are some cases in which this approach may not work: (1) when the team
generates new ideas during the selection process, as PPT assumes that all design
alternatives are understood a priori; (2) when the team size is large and an insufficient
proportion of team members do not have enough opportunities to express their ideas in
the team discussion; (3) when there are other mediums for data on preferences and
rationale other than pure discussion transcripts reflecting the team’s design thinking,
for example, the team uses many sketches rather than verbal “think aloud” in the
discussion for exchanging ideas; (4) the utterance of the design alternatives are
untraceable in a computational way, for example, the designers use many vague
words to represent the alternatives they mean in the discussion.
7.2 Future research
In transcript analysis of PPT, this research directly associates design choices of
the design alternatives with the utterances of the alternatives or the predefined
synonyms in the discussion. However, the unambiguous identification of a design
concept or alternative in text is an open area of research in linguistic. Lexical analysis
of the related words, such as synonyms, antonyms, hypernyms, hyponyms, meronyms,
holonyms, troponyms [37, 88] in the discussion transcript, could improve the accuracy
of language identification in the discussion transcript. In addition to the word
frequency, appraisal analysis [33] in the discussion also reflects designers’ preferences
in the design process. Integration of the appraisal analysis and lexical relationships are
to be developed in the future to improve the accuracy of the models for PPT.
137
In this work, while applying PPS, the maximum entropy is employed to construct
the rating distributions for the rating preferences. Exponential distribution is used
under the conservative way when a bounded rating value is given. In the future
research, more information such as individual weightings, group dynamic patterns is
to be employed to build more complicated distribution models.
In the comparative study in this study, periodic surveys are conducted through the
design process which may affect the team members’ discussion because of the
interruption. Audio and video recording equipments surrounded may also act as
confounding factors for the design process. Future research directs to consider how
much the confounding factors influence the experiment, and how to decrease the
distraction of the confounding factors.
138
Chapter 8 Contributions
The contributions on this research include:
z Design-specific text analysis on the transcribed discussion of the design
selection process. These text analysis methods can be extended to other
phases of engineering early stage design, such as requirements clarification
and concept generation.
z A probabilistic approach to extracting group preference information in
design selection stage. This approach is a potentially effective way to link the
qualitative design information that is generated by design teams with more
quantitative design decision making methods through the use of preference
information. It can work as an implicit alternative way for preference
extraction when the explicit surveys are not accessible or practical.
z A description of the design process in the form of preferential probability
evolution and alternative selection certainty. This process description method
could imply a viable way to measure and evaluate the design process. It is a
descriptive approach rather than a prescriptive approach which may lead a
novel way to understand the nature of a team’s preferences over time.
z Two probabilistic ways to convert preference ratings from surveys into
preferential probabilities which show the likelihood a design alternative is
most preferred by the team. The approaches link two forms of preference
139
information and may enlighten the research on converting the preferential
probabilities to the traditional preference ratings.
z Analysis and comparison between theoretically extracted preference
information and experimentally elicited preferences. The consistent results
between transcripts and the surveys in the case studies of this research
experimentally validate the effectiveness of the preferential probabilities
extracted from the transcripts. The way for comparing two sources of
preferential probabilities can be extended further for studying and analyzing
the similarity and identity for other research.

140
References
[1] Altshuller, G., Innovation Algorithm. 1973, Worcester, MA: Technical Innovation
Center.
[2] Altshuller, G., Creativity as an Exact Science. 1984, New York, NY: Gordon &
Breach.
[3] Arrow, K.J., Social Choice and Individual Values. 2nd ed. 1970, New Haven, CT,
US: Yale University Press. 126.
[4] Arrow, K.J. and H. Raynaud, Social Choice and Multicriterion Decision-Making.
1986, Cambridge, MA, US: The MIT Press.
[5] Atman, C.J. and K.M. Bursic, Verbal protocol analysis as a method to document
engineering student design processes. Journal of Engineering Education, 1998.
87(2): p. 121-132.
[6] Atman, C.J., J.R. Chimka, and K.M. Bursic. Results from a verbal protocol study
of the design process. in Frontiers in Education Conference, 27th Annual
Conference, Teaching and Learning in an Era of Change. 1997. Pittsburgh,
PA, USA.
[7] Baron, J. Value Trade-offs and the Nature of Utility: Bias, Inconsistency, Protected
values, and Other problems. in Conference on Behavioral Economics and
Neoclassical Economics, American Institute for Economic Research. 2002.
Great Barrington, MA, USA.
[8] Basak, I. and T.L. Saaty, Group decision making using the analytic hierarchy
process. Mathematical and computer modelling, 1993. 17(4-5): p. 101-109.
[9] Ben-Akiva, M. and S.R. Lerman, Discrete Choice Analysis. 1985, Cambridge,
Massachusetts: The MIT Press.
[10] Berelson, B., Content Analysis, in Handbook of social psychology, G. Lindzey,
Editor. 1954, Addison-Wesley: Reading, Massachusetts, USA. p. 488-522.
[11] Bertrand, M. and S. Mullainathan, Do People Mean What They Say? Implications
for Subjective Survey Data. The American Economic Review, 2001. 91(2): p.
67-72.
141
[12] Bierlaire, M., Discrete Choice Models, in Operations Research and Decision Aid
Methodologies in Traffic and Transportation Management, M. Labbe, et al.,
Editors. 1998, Springer Verlag. p. 203-227.
[13] Bilmes, J.A., A Gentle Tutorial of the EM Algorithm and its Application to
Parameter Estimation for Gaussian Mixture and Hidden Markov Models.
Technical Report. 1998, International Computer Science Institute: Berkeley,
CA, USA.
[14] Bockenholt, U., A Thurstonian Analysis of Preference Change. Journal of
Mathematical Psychology, 2002. 46(3): p. 300-314.
[15] Brans, J.P. and P. Vincke, A Preference Ranking Organisation Method: (The
PROMETHEE Method for Multiple Criteria Decision-Making). Management
Science, 1985. 31(6): p. 647-656.
[16] Breese, J.H., D. and C. Kadie. Empirical analysis of predictive algorithms for
collaborative filtering. in the Fourteenth Annual Conference on Uncertainty in
Artificial Intelligence. 1998: Morgan Kaufmann Publishers.
[17] Brockman, J.B. Evaluation of Student Design Processes. in The 26th Annual
Frontiers in Education Conference. 1996. Salt Lake City, Utah, USA.
[18] Burton, R.R., Semantic grammar: an engineering technique for constructing
natural language understanding systems. ACM SIGART Bulletin,
DEPARTMENT: Natural language interfaces, 1977(61): p. 26-26.
[19] Busemeyer, J.R. and A. Diederich, Survey of Decision Field Theory.
Mathematical Social Sciences, 2002. 43(3): p. 345-370.
[20] Canny, J. Collaborative filtering with privacy via factor analysis. in the 25th
annual international ACM SIGIR conference on Research and development in
information retrieval. 2002. Tampere, Finland: ACM Press.
[21] Chakrabarti, A. and T.P. Bligh, Approach to functional synthesis of mechanical
design concepts: theory, applications, and emerging research issues. Artificial
Intelligence for Engineering Design, Analysis and Manufacturing, 1996. 10(4):
p. 313-331.
[22] Chiu, I. and L.H. Shu. Bridging Cross-Domain Terminology for Biomimetic
Design. in ASME 2005 International Design Engineering Technical
Conferences & Computers and Information in Engineering Conference. 2005.
Long Beach, California, USA.
142
[23] Cronbach, L.J., Response Sets and Test Validity. Educational and Psychological
Measurement, 1946. 6: p. 475-495.
[24] Cronbach, L.J., Further Evidence on Response Sets and Test Designs.
Educational and Psychological Measurement, 1950. 10: p. 3-31.
[25] Cross, N., H. Christiaans, and K. Dorst, Analysing Design Activity. 1996,
Chichester: Wiley.
[26] Danielson, W.A. and D.L. Lasorsa, Perceptions of Social Change: 100 Years of
Front-Page Content in the New York Times and the Los Angeles Times, in Text
Analysis for the Social Sciences: Methods for Drawing Statistical Inferences
from Texts and Transcripts, C.W. Roberts, Editor. 1997, Lawrence Erlbaum
Associates Inc: Mahwah, New Jersey. p. 9-31.
[27] Delquie, P., Inconsistent Trade-offs between Attributes: New Evidence in
Preference Assessment Biases. Management Science, 1993. 39(11): p. 1382-
1395.
[28] Dempster, A., N. Laird, and D. Rubin, Maximum likelihood from incomplete data
via the EM algorithm. Journal of the Royal Statistical Society, Series B, 1977.
39(1): p. 1-38.
[29] Deshpande, M. and G. Karypis, Item-based top-N recommendation algorithms.
ACM Transactions on Information Systems, 2004. 22(1): p. 143-177.
[30] Dik, S.C. and K. Hengeveld, The Theory of functional grammar, Part 1: The
structure of the clause. 1991: Walter de Gruyter.
[31] Dong, A., The Latent Semantic Approach to Studying Design Team
Communication. Design Studies, 2005. 26(5): p. 445-461.
[32] Dong, A., Concept Formation as Knowledge Accumulation: A Computational
Linguistics Study. Artificial Intelligence for Engineering Design, Analysis and
Manufacturing, 2006. 20(1): p. 35-53.
[33] Dong, A., How Am I Doing? The Language of Appraisal in Design, in Design
Computing and Cognition '06 (DCC06), J.S. Gero, Editor. 2006, Kluwer:
Dordrecht, The Netherlands. p. 385-404.
[34] Dong, A., The Enactment of Design Through Language. Design Studies, 2007.
28(1): p. 5-21.
143
[35] Dong, A., A.W. Hill, and A.M. Agogino, A Document Analysis Method for
Characterizing Design Team Performance. Journal of Mechanical Design,
2004. 126(3): p. 378-385.
[36] Dong, A. and D. Mclnnes. Exploring the Relationship between Lexical Behavior
and Concept Formation in Design Conversations. in ASME 2005 International
Design Engineering Technical Conferences & Computers and Information in
Engineering Conference. 2005. Long Beach, CA, US.
[37] Dong, A., D. Mclnnes, and K.P. Davies. Exploring the Relationship between
Lexical Behavior and Concept Formation in Design Conversations. in ASME
2005 International Design Engineering Technical Conferences & Computers
and Information in Engineering Conference. 2005. Long Beach, CA, US.
[38] Dunning, T., Accurate methods for the statistics of surprise and coincidence.
Computational Linguistics, 1994. 19(1): p. 61-74.
[39] Dym, C.L., W.H. Wood, and M.J. Scott, Rank ordering engineering designs:
pairwise comparison charts and Borda counts. Research in Engineering
Design, 2002. 13(4): p. 236-242.
[40] Ericsson, K.A. and H.A. Simon, Protocol Analysis: Verbal Reports as Data.
1993, Cambridge, MA: MIT Press.
[41] Erisson, K.A. and H.A. Simon, Protocol Analysis: Verbal Reports as Data. 1993,
Cambridge, MA: MIT Press.
[42] Fishburn, P.C., Choice Probabilities and Choice Functions. Journal of
Mathematical Psychology, 1978. 18: p. 205-219.
[43] Fisher, R.A., On the mathematical foundations of theoretical statistics.
Philosophical Transactions of the Royal Society, 1922. 222: p. 309-368.
[44] Franzosi, R., One Hundred Years of Strike Statistics: Methodological and
Theoretical Issues in Quantitative Strike Research. Industrial and Labor
Relations Review, 1989. 42(3): p. 348-362.
[45] Franzosi, R., The Puzzle of Strikes: Class and State Strategies in Postwar Italy.
1995, Cambridge: Cambridge University Press.
[46] Gigone, D. and R. Hastie, The Impact of Information on Small Group Choice.
Journal of personality and social psychology, 1997. 72(1): p. 132-140.
144
[47] Gottsch, L.A., Content Analysis of Verbal Behavior: New Findings and Clinical
Applications. 1995, Hillsdale, New Jersey, USA: LAWRENCE ERLBAUM
ASSOCIATES.
[48] Green, P.E. and V. Srinivasan, Conjoint Analysis in Marketing: New
Developments with Implications for Research and Practice. Journal of
Marketing, 1990. 54(4): p. 3-19.
[49] Grefenstette, G. Automatic thesaurus generation from raw text using knowledge-
poor techniques. in The 9th Annual Conference of the UW Centre for the New
OED and text Research. 1993. Oxford, England.
[50] Halliday, M.A.K., An Introduction to Functional Grammar. 2nd ed. 1994,
London: A Hodder Arnold Publication.
[51] Hanley, N., S. Mourato, and R.E. Wright, Choice Modelling Approaches: A
Superior Alternative For Environmental Valuation? Journal of Economic
Surveys, 2001. 15(3): p. 435-462.
[52] Hauser, J.R. and D. Clausing, The House of Quality. Harvard Business Review,
1988. 66(3): p. 63-73.
[53] Hazelrigg, G.A., A framework for decision-based engineering design. Journal of
mechanical design, 1998. 120(4): p. 653-658.
[54] Hearst, M.A., Clustering versus faceted categories for information exploration.
Communications of the ACM, 2006. 49(4): p. 59-61.
[55] Hensher, D.A. and L.W. Johnson, Applied Discrete Choice Modeling. 1981, New
York: Halsted Press.
[56] Herlocker, J.L., et al. An algorithmic framework for performing collaborative
filtering. in the 22nd annual international ACM SIGIR conference on Research
and development in information retrieval. 1999. Berkeley, California, USA.
[57] Hey, J.D., Do Rational People Make Mistakes?, in Game Theory, Experience,
Rationality, W. Leinfellner and E. Kohler, Editors. 1998, Kluwer Academic
Publishers: Netherlands. p. 55-66.
[58] Hofmann, T., Latent semantic models for collaborative filtering. ACM
Transactions on Information Systems, 2004. 22(1): p. 89-115.
[59] Holsti, O.R., Content Analysis for the Social Sciences and Humanities. 1969,
Reading, Massachusetts, USA: Addison-Wesley.
145
[60] Howard, R.A. and J.E. Matheson, The Principles and Applications of Decision
Analysis. 1984, Menlo Park, CA: Strategic Decision Group.
[61] HumanMetrics, Jung Typology Test, http://www.humanmetrics.com/cgi-
win/JTypes2.asp.
[62] Jabeur, K., J.-M. Martel, and S.B. Khelifa. A Group Multicriterion Aggregation
Procedure Integrating the Relative Importance of Members. in 15th
International Conference on Multiple Criteria Decision Making. 1999. Ankara,
Turkey.
[63] Jabeur, K., J.-M. Martel, and S.B. Khelifa, A Distance-Based Collective Preorder
Integrating the Relative Importance of the Group's Members. Group Decision
and Negotiation, 2004. 13(4): p. 327-349.
[64] Jain, V.K. and D.K. Sobek, II, Linking Design Process to Customer Satisfaction
Through Virtual Design of Experiments. Research in Engineering Design,
2006. 17(2): p. 59-71.
[65] Jaynes, E.T., Information Theory and Statistical Mechanics. Physical Review,
1957. 106(4): p. 620-630.
[66] Jaynes, E.T., Prior Probabilities. IEEE Transactions On Systems Science and
Cybernetics, 1968. sec-4(3): p. 227-241.
[67] Jin, Y., D. Kim, and M.R. Danesh. Value based design: an objective structuring
approach to design concept generation. in ASME 2006 International Design
Engineering Technical Conferences & Computers and Information in
Engineering Conference. 2006. Philadelphia, Pennsylvania, USA.
[68] Jung, C.G., Psychological Types. Collected Works of C.G. Jung, ed. G. Adler and
R.F.C. Hull. Vol. 6. 1971: Princeton University Press.
[69] Keeney, R.L., A Group Preference: Axiomatization with Cardinal Utility.
Management Science, 1976. 23(2): p. 140-145.
[70] Keeney, R.L. and H. Raiffa, Decisions with Multiple Objectives: Preferences and
Value Tradeoffs. 1976, New York: Wiley.
[71] Kelley, C.T., Solving Nonlinear Equations with Newton's Method. 2003,
Philadelphia: Society for Industrial and Applied Mathematics.

146
[72] Kitamura, Y., et al. An Ontology-Based Annotation Framework for Representing
the Functionality of Engineering Devices. in ASME 2006 International Design
Engineering Technical Conferences & Computers and Information in
Engineering Conference. 2006. Philadelphia, Pennsylvania, USA.
[73] Kohrs, A. and B. Merialdo, Using category-based collaborative filtering in the
ActiveWebMuseum. the 2000 IEEE International Conference on Multimedia
and Expo, 2000. 1: p. 351-354.
[74] Krantz, D.H., et al., Foundations of Measurement Volume 1. Vol. I. 1971, New
York, NY, US: Academic Preess.
[75] Kulok, M. and K. Lewis, Preference Consistency in Multiattribute Decision
Making. ASME Conference Proceedings, 2005. 2005(4742Xa): p. 291-300.
[76] Landauer, T.K. and S.T. Dumais, A solution to plato's probem: The latent
semantic analysis theory of acquisition, induction, and representation of
knowledge. Psychological Review, 1997. 104(2): p. 211-240.
[77] Lawson, B., How Designers Think: The Design Process Demystified. 1998:
Butterworth Architecture.
[78] Lee, L., On the effectiveness of the skew divergence for statistical language
analysis, in Artificial Intelligence and Statistics. 2001. p. 65-72.
[79] Lewis, K. and F. Mistree, Modeling Interactions in Multidisciplinary Design - A
Game Theoretic Approach. AIAA Journal of Aircraft, 1997. 35(8): p. 1387-
1392.
[80] Lewis, K.E., W. Chen, and L.C. Schmidt, eds. Decision Making in Engineering
Design. 2006, ASME Press: New York, NY, USA.
[81] Li, W. and Y. Jin. Fuzzy Preference Evaluation for Hierarchical Co-Evolutionary
Design Concept Generation. in ASME 2006 International Design Engineering
Technical Conferences & Computers and Information in Engineering
Conference. 2006. Philadelphia, Pennsylvania, USA.
[82] Liu, Y.-C., T. Bligh, and A. Chakrabarti, Towards an 'ideal' Approach for
Concept Generation. Design Studies, 2003. 24(4): p. 341-355.
[83] Luce, R.D., Individual Choice Behavior. 1959, New York: Wiley.
[84] Mabogunje, A. and L.J. Leifer, 210-NP: Measuring the Mechanical Engineering
Design Process. Frontiers in Education Conference, 1996. FIE '96. 26th
Annual Conference., Proceedings of, 1996. 3: p. 1322-1328.
147
[85] Manski, C.F., The Structure of Random Utility Models. Theory and Decision,
1977. 8: p. 229-254.
[86] Markoff, J., Allies and Opponents: Nobility and Third Estate in the Spring of
1789. American Sociological Review, 1988. 53(4): p. 477-496.
[87] Miller, G.A., WordNet: A Lexical Database for English. Communications of the
ACM, 1995. 38(11): p. 39-41.
[88] Miller, G.A., et al., Introduction to WordNet: An On-line Lexical Database.
International Journal of Lexicography, 1990. 3(4): p. 235-244.
[89] Myers, J.L. and A.D. Well, Research Design and Statistical Analysis. 2nd edition
ed. 2003: Lawrence Erlbaum. 508.
[90] Nijstad, B.A., W. Stroebe, and H.F.M. Lodewijkx, Production blocking and idea
generation: Does blocking interfere with cognitive processes? Journal of
Experimental Social Psychology, 2003. 39: p. 531-548.
[91] Osborn, A.F., Applied imagination: Principles and procedures of creative
problem solving. 3rd ed. 1963, New York, NY: Charles Scribner's Sons.
[92] Otto, K.N. and E.K. Antonsson, Trade-Off Strategies in Engineering Design.
Research in Engineering Design, 1991. 3(2): p. 87-104.
[93] Otto, K.N. and E.K. Antonsson. The Method of Imprecision Compared to Utility
Theory for Design Selection Problems. in ASME 1993 Design Theory and
Methodology Conference. 1993.
[94] Packard, D.J., Preference Relations. Journal of Mathematical Psychology, 1979.
19(3): p. 295-306.
[95] PAHL, G. and W. BEITZ, Engineering design: A systematic approach. 1988,
New York: Springer-Verlag.
[96] Papalambros, P.Y. and D.J. Wilde, Principles of Optimal Design: Modeling and
Computation. 2nd ed. 2000, Cambridge: Cambridge University Press.
[97] Phadke, M.S., Quality Engineering Using Robust Design. 1995, Upper Saddle
River, NJ, USA: Prentice Hall PTR.
[98] Press, W.H., et al., Numerical Recipes: The Art of Scientific Computing. Third
Edition ed. 2007, New York: Cambridge University Press. 1256.
148
[99] Pugh, S., Total design: integrated methods for successful product engineering.
1991, Wokingham, England: Addison-Wesley.
[100] Riloff, E., An empirical study of automated dictionary construction for
information extraction in three domains. Artificial Intelligence, 1996. 85(1-2):
p. 101-134.
[101] Roberts, C.W., A Generic Semantic Grammar for Quantitative Text Analysis:
Applications to East and West Berlin Radio News Content from 1979.
Sociological Methodology, 1997. 27: p. 89-129.
[102] Roberts, C.W., ed. Text Analysis for the Social Sciences: Methods for Drawing
Statistical Inferences from Texts and Transcripts. 1997, Lawrence Erlbaum
Associates Inc: Mahwah, New Jersey, USA.
[103] Ross, S.M., Simulation. Fourth Edition ed. 2006, Burlington, MA: Academic
Press. 312.
[104] Saaty, T.L., Fundamentals of Decision Making and Priority Theory With the
Analytic Hierarchy Process. Analytic Hierarchy Process Series. Vol. 6. 2000,
Pittsburgh: RWS Publications.
[105] Sarwar, B., et al. Item-based collaborative filtering recommendation algorithms.
in the 10th international conference on World Wide Web. 2001. Hong Kong.
[106] Scott, M.J. and E.K. Antonsson, Compensation and Weights for Trade-offs in
Engineering Design: Beyond the Weighted Sum. Journal of Mechanical
Design, 2005. 127(6): p. 1045-1055.
[107] See, T.-K. and K. Lewis, A Formal Approach to Handling Conflicts in
Multiattribute Group Decision Making. Journal of Mechanical Design, 2006.
128(4): p. 678-688.
[108] Shah, J.J., et al., Collaborative Sketching (C-Sketch) - An Idea Generation
Technique for Engineering Design. Journal of Creative Behavior, 2001. 35(3):
p. 168-198.
[109] Shannon, C.E., A Mathematical Theory of Communication. Bell System
Technical Journal, 1948. 27(3): p. 379-423 & 623-656.
[110] Shapiro, G. and J. Markoff, A Matter of Definition, in Text Analysis for the
Social Sciences: Methods for Drawing Statistical Inferences from Texts and
Transcripts, C.W. Roberts, Editor. 1997, Lawrence Erlbaum Associates Inc:
Mahwah, New Jersey. p. 9-31.
149
[111] Shapiro, G. and J. Markoff, Revolutionary Demands: A Content Analysis of the
Cahiers de Doleances. 1998: Stanford University Press.
[112] Simpson, T.W., et al. Metrics for Assessing Design Freedom and Information
Certainty in the Early Stages of Design. in The 1996 ASME Design
Engineering Technical Conferences and Computers in Engineering Conference.
1996. Irvine, California, USA.
[113] Singh, V., et al. Innovations in Design Through Transformation: A Fundamental
Study of tRaNsFoRmAtIoN Principles. in ASME 2006 International Design
Engineering Technical Conferences & Computers and Information in
Engineering Conference. 2006. Philadelphia, Pennsylvania, USA.
[114] Song, S., A. Dong, and A.M. Agogino. Time Variation of Design "Story
Telling" in Engineering Design Teams. in Proceedings of the 14th
International Conference on Engineering Design (ICED 03). 2003. Stockholm,
Sweden.
[115] Suh, N.P., Axiomatic Design: Advances and Applications. 2001, Oxford
University Press. p. 39-53.
[116] Thompson, L.L., Making the Team: A Guide for Managers. 2nd ed. 2003, Upper
Saddle River, NJ: Prentice Hall.
[117] Thurston, D., A Formal Method for Subjective Design Evaluation with Multiple
Attributes. Research in Engineering Design, 1991. 3(2): p. 105-122.
[118] Thurston, D. and A. Locasio. Multi-attribute optimal design of structural
dynamic systems. in the 1992 ASME Design Theory and Methodology
Conference. 1992. Scottsdale, AZ, USA.
[119] Tribus, M., Rational Descriptions, Decisions, and Designs. 1969, New York:
Pergamon Press.
[120] Turney, P.D. Mining the Web for synonyms: PMI-IR versus LSA on TOEFL. in
The 12th European Conference on Machine Learning (ECML 2001). 2001.
Freiburg, Germany.
[121] Ueda, N. and R. Nakano, Deterministic annealing EM algorithm. Neural
Networks, 1998. 11(2): p. 271-282.
[122] Ueda, N., et al., SMEM Algorithm for Mixture Models. Neural Computation,
2000. 12(9): p. 2109-2128.
[123] Ullman, D.G., Mechanical Design Process. 3rd ed. 2003: McGraw-Hill.
150
[124] Ulrich, K.T. and S.D. Eppinger, Product Design and Development. 1995, New
York: McGraw-Hill, Inc.
[125] von Neumann, J. and O. Morgenstern, Theory of Games and Economic
Behaviour. 2nd Edition ed. 1947, Princeton: Princeton University Press.
[126] Vroom, V. and A. Jago, The New Leadership: Managing Participation in
Organizations. 1988, Upper Saddle River, NJ: Prentice-Hall.
[127] Vygotsky, L., Thought and Speech. 1986, Cambridge, MA: MIT Press.
[128] Wan, J. and S. Krishnamurty, Learning-Based Preference Modeling in
Engineering Design Decision-Making. Journal of Mechanical Design, 2001.
123(2): p. 191-198.
[129] Wang, J., A Fuzzy Outranking Method for Conceptual Design Evaluation.
International Journal of Production Research, 1997. 35(4): p. 995-1010.
[130] Wang, J., A.P. de Vries, and M.J.T. Reinders. Unifying User-based and Item-
based Collaborative Filtering Approaches by Similarity Fusion. in the 29th
annual international ACM SIGIR conference on Research and development in
information retrieval. 2006. Seattle, Washington, USA.
[131] Wassenaar, H.J. and W. Chen, An Approach to Decision Based Design with
Discrete Choice Analysis for Demand Modeling. Journal of Mechanical
Design, 2003. 125(3): p. 490-497.
[132] Wassenaar, H.J., et al., Enhancing Discrete Choice Demand Modeling for
Decision-Based Design. Journal of Mechanical Design, 2005. 127(4): p. 514-
523.
[133] Wikipedia, Taxicab geometry, http://en.wikipedia.org/wiki/Taxicab_geometry.
[134] Witherell, P., S. Krishnamurty, and I. Grosse. Ontologies for Supporting
Engineering Design Optimization. in ASME 2006 International Design
Engineering Technical Conferences & Computers and Information in
Engineering Conference. 2006. Philadelphia, Pennsylvania, USA.
[135] Wood, K.L. and E.K. Antonsson, Computations with Imprecise Parameters in
Engineering Design: Background and Theory. ASME Journal of Mechanisms,
Transmissions, and Automation in Design, 1989. 111(4): p. 616-625.
[136] Yang, M.C., Concept Generation and Sketching: Correlations With Design
Outcome. ASME Conference Proceedings, 2003. 2003(37017b): p. 829-834.
151
[137] Yang, M.C., A Study of Prototypes, Design Activity, and Design Outcome.
Design Studies, 2005. 26(6): p. 649-669.
[138] Yang, M.C. and H. Ji, A Text-based Analysis Approach to Representing the
Design Selection Process. Proceedings of the 16th International Conference on
Engineering Design (ICED 07), 2007.
[139] Yang, M.C., W. Wood, and M. Cutkosky, Design information retrieval: a
thesauri-based approach for reuse of informal design information. Engineering
with Computers, 2005. 21(2): p. 177-192.
[140] Yang, M.C., W.H. Wood, and M.R. Cutkosky, Design Information Retrieval: A
Thesauri-based Approach for Reuse of Informal Design Information.
Engineering with Computers, 2005. 21(2): p. 177-192.

152
Appendix A: Surveys Used in Coffee Maker Re-design Experiment
Question 1: Based on your own opinion and on the discussion so far, how would you
rank the following three alternatives for the carafe selection problem? If you have 10
points totally, how would you allocate these points on the following three alternatives,
with larger number meaning more preference?

Name/ID Glass coffee pot
Glass coffee carafe
Coffee pot A
Carafe A
Steel coffee pot
Steel coffee carafe
Stainless-steel
carafe
Coffee pot B
Carafe B
Plastic coffee pot
Plastic coffee
carafe
Coffee pot C
Carafe C
Photo

Rank

Rating (sum to
10 points total)

Rationale (the
simple reason
of your
selection)

153

Question 2: How would you rank the following three alternatives for the filter
selection problem? If you have 10 points totally, how would you allocate these points
on the following three alternatives, with larger number meaning more preference?

Name/ID Gold tone filter
Filter A
Paper filter
Disposable filter
Filter B
Titanium filter
Ti filter
Filter C
Photo

Rank

Rating (sum to
10 points total)

Rationale (the
simple reason
of your
selection)

154
Appendix B: A Segment Sample from Discussion Transcripts of
Coffee Maker Re-design (in XML File)
……………..
-
16:00
H
Glass Coffee carafe seems to have the most capacity.

-
16:10
I
" For the coffee pot A, the capacity says that it can be designed as wanted,
available for 2 cups and 6 cups. So at least for this option A (glass pot), it has a lot
of flexibility."

-
16:25
P
" It is the same for the steel coffee pot, it says the same thing"

-
16:32
H
The same for the plastic (plastic coffee pot)

-
16:38
I
So it is not a constraint at all. We do not have to think about it.

-
16:43
P
" OK, forget about capacity. Then what about weight? Because if you see the
requirement, it says you are in good health, but you are not as strong or mobile
as you were when you were younger. So maybe it's an old person, and he or she
cannot deliver very heavy coffee pots."

155
-
16:54
H
That's the weight and portability.

-
16:58
P
" Yes, weight and portability, I think important factors. "

-
17:02
H
" One thing here is another following is it seems it is true they can provide in 2
cups to 6 cups. But according to their design, you see that the stainless-steel
carafe, if we want to have 6 cups (from) the stainless-steel, it will be provided in
very big, heavy as supposed to the plastic one. In plastic, we have only a single
thing."

-
17:30
I
" Yes, we should exchange the information about the weight and the
portability. And for the coffee pot A, The weight is light, and the portability says
not portable. I do not quite understand what does it mean by not
portable."

-
18:07
P
" I think portability means the ability to shift from one place to another.
Maybe because it is glass, it might break. "

-
18:13
I
" OK, and how about..."

-
18:15
P
" And for the steel coffee pot, coffee pot B, it says it is heavy, but portability is
portable. So probably you can take it with you if you..."
156

-
18:20
H
So there is a drawback for stainless-steel because it is heavy.

-
18:22
P
" It is heavy, exactly."

-
18:25
I
Then how about for the plastic pot?

-
18:28
H
" the Plastic, it is portable, it is light, and not easy to clean, not attractive, and
less durable, footprint size is small, and fragile material inside, fragility, and $15
price cost, and thermal-insulated plastics."

-
18:40
P
so the cost is $15?

-
18:44
H
costing is $15.

-
18:50
P
And what about warming plate cost? 0?

-
18:56
H
" 0, yeah."

157
-
19:02
P
" OK, and it can keep the coffee warm because it is thermal-insulated."

-
19:07
H
" Yes, thermal-insulated inside, but outside is fragile. So..."

-
19:12
P
" So the thing is..., what is your durability for the glass coffee pot, Iris?"

-
19:18
I
" Durability (of glass pot), it says durable, reliable."

-
19:24
H
Because they usually they use temper glass.

-
19:27
P
Ok.

-
19:28
H
They (glass pot) are not fragile (?).

-
19:30
P
So I think...

-
19:32
I
158
So what is the cost for the steel (steel pot)?

-
19:33
P
It is $20.

-
19:34
I
$20.00

-
19:35
H
So more costly.

-
19:36
I
and warming plate is 0?

-
19:38
P
" yes. (Steel pot) Warming plate is 0, and footprint size is small, and fragility is
strong, it does not break, durability is durable, heat retention is OK with double
layers of steel, and weight is very heavy, and portability is portable, and not easy
to clean"

-
20:06
[The second questionnaire]

-
22:12
H
So is it (glass pot) attractive (?)

-
22:15
I
159
I do not have the information for the plastic coffee pot. Is it easy to
clean?

-
22:25
H
" No, it is not easy to clean. It's not attractive. Can be designed as wanted,
available for 2 cups and 6 cups."

-
22:35
I
So how about the steel coffee pot? How about the style and aesthetic value? Is
it looking attractive?

-
22:45
P
" Yes, it looks attractive."

……………..

Abstract (if available)

Abstract Activities in the early stage of engineering design typically include the generation of design choices and selection among these design choices. A key notion in design alternative selection is that of preference in which a designer or design team assigns priorities to a set of design choices. However, preferences become more challenging to assign on both a practical and theoretical level when done by a group of individuals. Preferences may also be explicitly obtained via surveys or questionnaires in which designers are asked to rank the choices, rate choice with values, or select a "most-preferred" choice. However, these methods are typically employed at a single point of time

Linked assets

University of Southern California Dissertations and Theses

Conceptually similar

PDF

A synthesis reasoning framework for early-stage engineering design

PDF

Using organized objectives to structure arguments for collaborative negotiation of group decisions in software design

PDF

Managing functional coupling sequences to reduce complexity and increase modularity in conceptual design

PDF

A social-cognitive approach to modeling design thinking styles

PDF

Designing‐in performance: energy simulation feedback for early stage design decision making

PDF

Decision support systems for adaptive experimental design of autonomous, off-road ground vehicles

PDF

AI-driven experimental design for learning of process parameter models for robotic processing applications

PDF

Design and analysis of reduced complexity transceivers for massive MIMO and UWB systems

PDF

Proactive detection of higher-order software design conflicts

PDF

Hashcode representations of natural language for relation extraction

PDF

Developing an agent-based simulation model to evaluate competition in private health care markets with an assessment of accountable care organizations

Asset Metadata

Creator Ji, Haifeng (author)

Core Title Extraction of preferential probabilities from early stage engineering design team discussion

School Viterbi School of Engineering

Degree Doctor of Philosophy

Degree Program Industrial and Systems Engineering

Publication Date 10/07/2008

Defense Date 08/19/2008

Publisher University of Southern California (original), University of Southern California. Libraries (digital)

Tag concept selection,design decision-making,design process,OAI-PMH Harvest,preferences,probabilities

Language English

Contributor Electronically uploaded by the author (provenance)

Advisor Yang, Maria C. (committee chair), Jin, Yan (committee member), Lu, Stephen (committee member)

Creator Email haifeng.ji@gmail.com,haifengj@usc.edu

Permanent Link (DOI) https://doi.org/10.25549/usctheses-m1635

Unique identifier UC173152

Identifier etd-Ji-2413 (filename),usctheses-m40 (legacy collection record id),usctheses-c127-120085 (legacy record id),usctheses-m1635 (legacy record id)

Legacy Identifier etd-Ji-2413.pdf

Dmrecord 120085

Document Type Dissertation

Rights Ji, Haifeng

Type texts

Source University of Southern California (contributing entity), University of Southern California Dissertations and Theses (collection)

Repository Name Libraries, University of Southern California

Repository Location Los Angeles, California

Repository Email cisadmin@lib.usc.edu