Validation of Design Tools-PPT for CEDAR Meeting-04-15-2016

CED RClemson Engineering Design Applications and Research
Varun Kumar
kumar3@clemson.edu
Committee:
Dr. Gregory Mocko (Chair)
Dr. Joshua Summers
Dr. George Fadel
Similarity assessment of design problems
used in creativity research

2 of 26
kumar3@clemson.eduCED RClemson Engineering Design Applications and Research
 Research background
 Research objectives
 Design problems in creativity research
 Design problem network
 Problem similarity assessment
– Based on structural elements
– Based on Latent Semantic Analysis
 Conclusions
 Other tasks completed
 Future work
 References
Outline

3 of 26
Research Overview
Study 1
Problem
1
Method
1
Metric 1Subject
Protocol
1
Study 2
Problem
2
Method
2
Metric 2Subject
Protocol
2
Evaluate the impact of different
methods on design creativity
- Various sources
of difference
between studies
are present
- What prohibits
comparison of two
methods based
on published
literature alone?
- Design problems
and their place in
creativity research

4 of 26
Title Author, Year
Cognitive processes and ill-defined problems: A case study
from design
Eastman, C. M., 1969
Dilemmas in a General Theory of Planning Rittel, H. W. J., & Webber, M.
M., 1973
The structure of ill-structured problems Simon, H. A., 1977
A Proposed Taxonomy of Mechanical Design Problems Dixon, J. R., 1988
The structure of design problem spaces Goel, V., and Pirolli, 1992
A Suggested Taxonomy for Engineering Design Problems Frost, R. B., 1994
Mechanical Engineering Design Complexity Metrics: Size,
Coupling, and Solvability
Summers, J. D., and Shah, J.
J., 2010
In search of effective design problems for design research Durand et al., 2015
Research background

5 of 26
Research opportunities
Conceptual
Problems
Use in protocol/
study new tool
or method
Analyze
creativity
metrics
Compare
problem similarity
based on metric
measures
Research Question 2
How to assess problem similarity based
solely on problem representation?
Summers and Shah’s approach Durand and Linsey’s approach
Opportunity
a. How to illustrate the application in
conceptual problems?
b. Can natural language
representation be used to compare
problems for similarity?
Opportunity
a. How to assess problem similarity
based solely on problem
representation?

6 of 26
 Understand the pattern of design problem usage.
 Enable similarity comparison between conceptual design problems.
 Evaluate the impact of problem size on between-study treatment effects.
Research Objectives

7 of 26
 Statement of requirement, needs, functions or objectives.
 Most problems are ill-defined and co-evolve with solutions.
 Conceptual design problems differ from real life design problems.
 Examples:
Design Problems in creativity research
Design Problem 1 (Linsey, 2012)
Design and build a low-cost, easy to
manufacture peanut shelling machine that
will increase the productivity of the African
peanut farmers. Target throughput is
approximately 50 Kg per hour. The goals
include: a) Must remove the shell with
minimal damage to peanuts b) Electrical
outlets are not available as a power source
c) A large quantity of peanuts must be
quickly shelled.
Design Problem 2 (Mulet, 2012)
It is asked to design a new table for
offices that allows alternate sitting and
stand up work. There are a lot of people
who must work on sitting position the full
day. The possibility to alternate positions
during working time could drive to an
improvement in health and productivity.
The current tables that allow combining
positions in work have limited surface, not
enough for design, architecture and
engineering needs.

8 of 26
Problem Usage Network
Problem
Author/s

9 of 26
Metric Value Conclusions
Network density 0.0751 Poor connectivity between problems or vice-versa (between
researchers)
No. of nodes with
degree 0 or 1
20* High percentage of problems which have not been re-used
No. of
communities
20** - Presence of sub-groups
- Problem sharing within sub-groups
Network Analysis
* Total nodes = 50
** Includes isolated nodes

10 of 26
 Lack of collaboration
 Absence of guidelines for formulating conceptual problems
 No ‘benchmark’ problems in practice
 Absence of methods to compare problems
 Impact of problems on creativity results not understood fully
Possible reasons for disconnectedness

11 of 26
 Why compare problems?
– Help reduce variability in structure of design problems
– Enable problem replacement in pre-post test experiment designs
– Provide metric for assessing the choice of problem
– Enable use of ‘similar’ problems in practice
 Methods for problem similarity assessment
– Approach 1: Identification of structural elements in problem
statement
– Approach 2: Latent Semantic analysis of problem statements
 Objective is to compare similarity based on problem representation
Assessment of problem similarity

12 of 26
Approach 1: Element identification
Design
Problem
Goals of the problem
Functional
Requirements
Non-functional
requirements
Information about end
user
Reference to an existing
product

13 of 26
 Goal of the problem
– Final objective of design task
– Problem may have more than one goal
– Ratio variable
 Functional requirements
– Primary functions of designed object
– Should not insinuate about method for achieving function
– Generally action verbs associated with objects
– Can be nouns derived from verbs (eg. washing machine)
 Non-functional requirements
– Do not describe primary function
– Determine shape, size, operation and design selection
Approach 1: Element definitions

14 of 26
 Information about end user
– Knowledge of end user in problem statement
– Should be explicitly stated
– Categorical variable (Yes/No)
 Reference to an existing product
– Whether design problem hints at an existing product known to
reader.
– May be a ‘conundrum’ due to cultural differences
– Categorical number (Yes/No)
Approach 1: Element definitions (contd.)

15 of 26
Approach 1: Element identification example 1

16 of 26
Approach 1: Element identification example 2

17 of 26
 Based on elements identified and their ‘counts’ problems can be
compared
Approach 1: Comparing two problems
Design Problem
Design Problem Element Example 1 Example 2
No. of goals 1 1
No. of Functional requirements 1 2
No. of non-functional requirements 2 4
End user info. (Yes=1/No=0) 1 1
Reference to an existing product
(Yes=1/No=0)
1 1

18 of 26
 Definition of elements always subject to subjective interpretation
 Test whether people identify the same elements and their quantities.
 Method used:
– 4 evaluators/raters chosen
– Evaluators are presented with definitions of the 5 elements
– An identification example is shown to all evaluators separately
– 4 evaluators asked to identify the 5 elements from 4 design problems
– Inter-rater agreement evaluated
Approach 1: Verifying similarity comparison protocol
Elements
No. of
goal
Functional
requirement
Non-
Functional
requirement
End user
info
Ref. to
existing
product
Krippendorff’s
alpha
1.0 1.0 0.863 0.184 0.598

19 of 26
 Pros
– Useful as a starting guide for problem formulation
– Offers useful and simple mean for comparing 2 problems
– Helps identify ‘information content’ of problem statements
– Method shows good correlation between evaluators for most
elements
– Compares problems based on representation
 Cons
– Subjectivity of human interpretation still involved
– Cannot compare ‘knowledge content’ and skill needed in problem
solving.
Approach 1: Conclusions

20 of 26
 Linguistic similarity of design problem representations
 Relies on extracting contextual meanings from large text corpus.
 Assumes that words and phrases with similar meaning are used in
similar context
 Generates similarity scores between -1 and 1
 LSA tool available at http://lsa.colorado.edu/ used
 45 design problem statements tested for LSA similarity
Approach 2: Latent Semantic Analysis

21 of 26
Approach 2: LSA results
Problem statements
Design an urban
bi or tri cycle
for use by white
collar workers
Design a concrete
mixer which can
operate using a
bicycle pedal
mechanism.
Design a device
which can
compost waste
vegetables.
Design a reading
device for old people
which can read the
newspaper for them.
Design a
water
lifting
device.
Propose alternative solution to coal
pile problem at thermal plant, since the
plant may not have enough land
nearby to store the coal on ground.
0.02 0.01 0.06 0.01 0.03
Design of a next generation alarm
clock which ensures easy operations
like change of time and alarm stop
unlike conventional clocks.
0.09 0.26 0.15 0.13 0.12
Design of a litter collection device for
volunteers.
0.23 0.48 0.6 0.25 0.39
Redesign an electric toothbrush for
increased portability.
0.01 0 -0.01 -0.01 0.03
Design alternative means to manually
propel boats which are easy to
maneuver, don't rock the boat or splash
water.
0.09 0.15 0.15 0.06 0.52
A portion of the LSA result matrix

22 of 26
 Pros
– Objective way of comparing two problem statements
– Based on problem representation only
– Backed by statistical computation, avoids subjectivity
– Can detect contextual changes between text bodies even with slight
modifications (viz. design vs redesign etc.)
 Cons
– Results only as good as the corpus of text used
– Based solely on representation, problem ‘solvability’ can’t be
compared
Approach 2: Conclusions

23 of 26
 Evaluate the impact of problem size on between-study treatment effects.
– Meta-Analysis of different studies to assess treatment effect.
– Analysis of heterogeneity between treatment effect of various studies
– Meta-regression approach to understand impact of problem size on
heterogeneity
Other works completed

24 of 26
Write Thesis
Defend
Enjoy Summer
Future work

25 of 26
 Linsey, J. S., Clauss, E. F., Kurtoglu, T., Murphy, J. T., Wood, K. L., and Markman, a. B., 2011, “An
Experimental Study of Group Idea Generation Techniques: Understanding the Roles of Idea
Representation and Viewing Methods,” J. Mech. Des., 133(3), p. 031008.
 Chulvi, V., Sonseca, A., Mulet, E., and Chakrabarti, A., 2012, “Assessment of the Relationships Among
Design Methods, Design Activities, and Creativity,” J. Mech. Des., 134(11), p. 111004.
 Ameri, F., Summers, J. D., Mocko, G. M., and Porter, M., 2008, “Engineering design complexity: An
investigation of methods and measures,” Res. Eng. Des., 19(2-3), pp. 161–179.
 Rittel, H. W. J., and Webber, M. M., 1973, “Dilemmas in a General Theory of Planning,” Policy Sci.,
4(December 1969), pp. 155–169.
 Simon, H. A., 1977, “The structure of ill-structured problems,” Models of discovery, Springer, pp. 304–
325.
 Dixon, J. R., Duffey, M. R., Irani, R. K., Meunier, K. L., Orelup, M. F., 1988, “A Proposed Taxonomy of
Mechanical Design Problems,” Comput. Eng., 1, pp. 41–46.
 Goel, V., and Pirolli, P., 1989, “Motivating the notion of generic design within information-processing
theory: the design problem space,” AI Mag., 10(1), pp. 18–36.
 Goel, V., and Pirolli, P., 1992, “The structure of design problem spaces,” Cogn. Sci., 16(3), pp. 395–
429.
 Frost, R. B., 1994, “A Suggested Taxonomy for Engineering Design Problems,” J. Eng. Des., 5(4), pp.
399–410.
 Summers, J. D., and Shah, J. J., 2010, “Mechanical Engineering Design Complexity Metrics: Size,
Coupling, and Solvability,” J. Mech. Des., 132(2), p. 021004.
 Tsenn, J., Helms, M. E., Linsey, J. S., and Mcadams, D. A., 2015, “In search of effective design
problems for design research,” pp. 1–10
References

CED RClemson Engineering Design Applications and Research
Questions?

27 of 26
Appendix A: Design problem used for experimental verification
of protocol
Problem 1: Design a new system for gathering together and hiding the wires of the electronic equipment in an
office table. Currently the work in the field of design, architecture and engineering needs of a personal computer,
printers, and scanners. Each of these devices needs of electrical supply and the wires on table surface are annoying.
Actually, there are simple solutions to gather them, but it is difficult to extract or introduce a wire, or they leave the
wires hanging behind the table.
Problem 2: Design an automatic recycler device that can automatically sort plastic bottles, glass containers,
aluminum cans, and tin cans. The major differentiation between types of materials lay with the given dimensions of
the products: plastic bottles are the tallest, glass containers are very short and heavy, and aluminum cans are
lightweight. Devices are given strict requirements to adhere to such as volume and weight constraints, safety
requirements, and most importantly, have to operate autonomously once a master shutoff switch is toggled.
Problem 3: A mechanical system is required which, in the event of a fire, will enable people to escape from a
six-storey building by lowering themselves to the ground from windows. The system, which might make use of a 5
mm diameter steel cable, must be capable of lowering either a small child or a heavy adult at approximately the
same constant speed.
Problem 4: Design and develop an artifact to facilitate grocery shopping in a typical French/Singapore
Chinatown fresh market. The artifact should facilitate carrying of groceries from fresh market to home.

28 of 26
S.No. Characteristic Definition Scoring system
1.
Reference to existing
product
Whether or not the problem statement contains a
reference to an existing product.
Questions to be asked
 Does the problem statement contain reference to
an existing product?
 Does the problem statement ask you to re-design
an existing product?
If a reference to existing product exists in the
problem statement, assign a score of 1, else 0
2.
Functional
requirements (FR)
These are the things which the product needs to do, or
the tasks that you want the product to perform without
any reference to how it should be done. Judgement
should be based only on explicitly stated texts, and not
on the implied meaning of a sentence. To identify FRs,
look for:
1. Action verbs like move, work etc. associated with
objects (objects include nouns on which the
action verbs act like throw stones, gather fruits
etc.)
2. Primary functions of the design (eg. move
objects, lift, transport etc.).
3. These could also be nouns derived from verbs
(eg. washing machine, toaster etc.)
4. If there are two objects associated with one
primary function, count it as 2 separate FRs (eg.
move object X & object Y is counted as 2 FRs)
 What are the primary functions of the product?
 What are the expected outputs/tasks which the
product needs to perform?
Count the number of functional requirements
given in the problem statement. There can be 2
cases:
1: When a new product design is desired: In
this case, FR should be specified in the
problem statement. Else, give a score of 0.
2: When a re-design or a new design for an
existing product is desired: FR count in this
case is already 1 to start off, since atleast 1
product function is known.
Appendix B: Element definitions

29 of 26
S.No. Characteristic Definition Scoring system
3.
Non-functional
requirements
These are 'non-functional' requirements, which do not determine
the primary functions of the product, but cast a bound on the
overall shape, size, cost, operation and selection of the design.
Judgement should be based only on explicitly stated texts, and not
on the implied meaning of a sentence. Typical NFRs include:
1. Any restrictions on what the product or system shall or shall
not do apart from the primary functions (eg. should not
overheat, should be easy to use, should be manufacturable
etc.)
2. Any restriction on how the product shall fulfil its intended
functions (eg. device should move using rollers, device
should work using sliding mechanism etc.)
3. Any qualities that the product must possess (eg. easy to
make, easy to use, cheap, etc.)
• What things cast a limit or a bound on the solution space?
 What are the qualities that the product should possess as a
whole?
 How the overall product 'shall be' like?
Count the number of Non-functional requirements
given in the problem statement.
4. Number of goals
These are goals or final objectives associated with the design
problem.
 What is the final objective of the problem statement, or
 Does the problem statement ask only to design or do
something else?
Count the number of goals or objectives mentioned
in the problem statement.
5.
Information about end
user
Information about who is going to use the product or who is the
customer. It should be explicitly stated in the problem statement.
 Who is going to be the end user of the product?
Check the problem statement to see if any
information about the end user is provided or not. If yes,
give a score of 1, else a 0
Appendix B: Element definitions

30 of 26
Authors
Eastman
(1969)
Rittel and
Webber
(1973)
Dixon
(1988)
Goel
(1992)
Frost
(1994)
Summer
s and Shah
(2010)
Durand
and Linsey
(2015)
 Lack of
well -defined
specifications
for goals
 Lack of
formal
representation
language
 Lack of
definitive
formulation
 Lack of
known solution
states
 Lack of
immediate tests
for solutions
 Lack of
objectivity in
solution
selection
 Lack of
learning
opportunities by
trial and error
 Lack of
exhaustive set of
solutions

Perceived
need
 Function

Phenomenon

Embodiment
 Artefact
type
 Artefact
instance
 Lack of
problem
structuring

Presence of
distinct
problem
solving
phases

Reversal of
direction of
transformatio
n functions

Modular
structure

Incremental
solution
transformatio
n

Personalized
stopping rules

Abstraction
hierarchies
 Type of
entity being
designed
 Degree of
innovation
involved
 Extent of
possible
decomposition of
designed entity
 Availability
of adaptable
solutions
 Complexity
of designed entity
 Degree of
interaction within
solution
 Looseness
or tightness of
constraints
 Number of
artefacts to be
built
 Number
of independent
design
variables
 Number
of dependent
design
variables
 Number
of design
relations
 Number
of measures of
goodness
 Number
of functional
requirements
and design
parameters
 Number
of design
constraints
 Size

Connectedness

Participant's
familiarity with
problem

Participant's
familiarity with
solution and
principles
 Size of
potential solution
space
 Assumed
constraints due to
known solutions
 Effort
required
 Fixation
 Problem
domain

31 of 26
DP1 Toy to transport basket
DP2 Subway improvement
DP3 New drawing table
DP4 Tubular map case
DP5
System to collect and
hide electronic wires
DP6 New table for offices
DP7 Rover device
DP8 Automatic recycler
DP9 Wearable binoculars
DP10 Bi/tri-cycle
DP11
Bicycled Concrete
Mixer
DP12 Vegetable Composter
DP13 Reading Device
DP14 Water lifting device
DP15 Traffic light using LED

32 of 26
DP16 Counting and packaging device
DP17 Milk frothing device
DP18 Peanut shelling machine
DP19 Automatic casting system
DP20 Robotic vacuum
DP21 Portable washing machine
DP22 Solar heating and cooking device
DP23 Rigless abandonment tool
DP24 Wheelchair simulator
DP25 Counter top stand
DP26 Split pin design
DP27 Device to transport ping pong ball
DP28 Tool for alien species
DP29 Biomass based cooking system
DP30 System for UAVs
DP31 Remote controller
DP32 Doodle toaster

33 of 26
DP33
Combined toaster and coffee
maker
DP34 Horizontal toaster
DP35 Crumb tray toaster
DP36 Shopping cart
DP37 Outdoor customer product
DP38 Manual boat propulsion device
DP39 Plastic and paper sorting device
DP40 Lunar dust protection device
DP41 Desk elevator
DP42 Ship guitar
DP43 Bottle cap regiser machine
DP44 Soda drink maker
DP45 Concept to center mating parts
DP46 Concept to use snow as insulator
DP47 Coal pile solution
DP48 Next gen alarm clock
DP49 Litter control device
DP50 Electric tooth brush

Validation of Design Tools-PPT for CEDAR Meeting-04-15-2016

More Related Content

What's hot (19)

Viewers also liked (18)

Similar to Validation of Design Tools-PPT for CEDAR Meeting-04-15-2016 (20)

Validation of Design Tools-PPT for CEDAR Meeting-04-15-2016

Editor's Notes