Introduction to Design Research:
Introduction to Design Research:
a Methodological Background for
a Methodological Background for
Scientific Work
Outline
Outline
Motivation
Motivation
Types of research
Types of research
Design Research Basics
Design Research Basics
Evaluation in Design Research
Evaluation in Design Research
Motivation
Motivation
Motivation for research:Motivation for research:
pure research pure research: enhance understanding of phenomena: enhance understanding of phenomena
instrumentalist researchinstrumentalist research: : a a problem needs problem needs a a solutionsolution
applied research: a solution needs application fieldsapplied research: a solution needs application fields
Motivation for research methodologyMotivation for research methodology
(qualitatively) control research process(qualitatively) control research process
validate research resultsvalidate research results
compare research approachescompare research approaches
Research: A Definition
Research: A Definition
Research:Research:
anan activity activity that contributes to thethat contributes to the understanding understanding of aof a
phenomenon
phenomenon [Kuhn, 1962; Lakatos, 1978][Kuhn, 1962; Lakatos, 1978]
phenomenon: aphenomenon: a set of behaviors of some entity set of behaviors of some entity (ies)(ies)
that is found
that is found interesting interesting by a research communityby a research community
understanding:understanding: knowledge that allows predictionknowledge that allows prediction of of
the behavior of some aspect of the phenomenon the behavior of some aspect of the phenomenon
activities considered appropriate to the production of activities considered appropriate to the production of
understanding (knowledge) are the research methods understanding (knowledge) are the research methods and techniques of a research community
and techniques of a research community
paradigmatic vs multi-paradigmatic communitiesparadigmatic vs multi-paradigmatic communities
(agreement on phenomena of interest and research (agreement on phenomena of interest and research methods)
Scientific Disciplines
Scientific Disciplines
Types of research [Simon, 1996]:Types of research [Simon, 1996]:
natural sciences: phenomena occurring in the worldnatural sciences: phenomena occurring in the world
(nature or society) (nature or society)
design sciences ~ sciences of the artificial:design sciences ~ sciences of the artificial:
all or part of the phenomena may be created artificiallyall or part of the phenomena may be created artificially
studies artificial objects or phenomena designed to meetstudies artificial objects or phenomena designed to meet certain goals
certain goals
social sciences: structural level processes of a socialsocial sciences: structural level processes of a social
system and its impact on social processes and social system and its impact on social processes and social organization
organization
behavioural sciences: the decision processes andbehavioural sciences: the decision processes and
communication strategies within and between organisms communication strategies within and between organisms in a social system
[Owen,1997] [Owen,1997] activities activities phenomena phenomena design design sciences sciences Semantic Web Semantic Web (CS) (CS)
Design research basics
Design research basics
Process model
Process model
Artifact types:
Artifact types:
result of the research workresult of the research work
Artifact structure
Artifact structure
content of the research approachcontent of the research approach
Evaluation:
Evaluation:
evaluation criteriaevaluation criteria
Process model
Process model
a problem-solving paradigm:
a problem-solving paradigm:
seeks to create innovations that define theseeks to create innovations that define the
ideas, practices, technical capabilities, and ideas, practices, technical capabilities, and products through which the analysis, design, products through which the analysis, design, implementation, and use of information
implementation, and use of information systems can be effectively and
systems can be effectively and efficientlyefficiently accomplished [Tsichritzis 1997; Denning accomplished [Tsichritzis 1997; Denning 1997]
[Takeda,1990]
[Takeda,1990]
Awareness of
Awareness of
problem
problem SuggestionSuggestion DevelopmentDevelopment EvaluationEvaluation ConclusionConclusion
a
a
b
b
d
d
u
u
c
c
t
t
i
i
o
o
n
n
d
d
e
e
d
d
u
u
c
c
t
t
i
i
o
o
n
n
+ operation and goal knowledge
+ operation and goal knowledge
circumscription
circumscription
knowledge knowledge flows flows process process steps steps logical logical formalism formalismDesign research process
Artifacts
Artifacts
are not exempt from natural laws or behavioralare not exempt from natural laws or behavioral
theories theories
artifacts rely on existing "kernel theories" thatartifacts rely on existing "kernel theories" that
are applied, tested, modified, and extended are applied, tested, modified, and extended through the experience, creativity, intuition, through the experience, creativity, intuition, and problem solving capabilities of the
and problem solving capabilities of the
researcher [Walls et al. 1992; Markus et al. researcher [Walls et al. 1992; Markus et al. 2002]
Design research outputs
Design research outputs
[March & Smith, 1995]
[March & Smith, 1995]
ConstructsConstructs
conceptual vocabulary of a problem/solution domainconceptual vocabulary of a problem/solution domain
MethodsMethods
algorithms and practices to perform a algorithms and practices to perform a specific taskspecific task
ModelsModels
a set of propositions or a set of propositions or statements expressingstatements expressing
relationships among
relationships among constructsconstructs
abstractions and representationsabstractions and representations
InstantiationsInstantiations
constitute the realization of constructs, models andconstitute the realization of constructs, models and
methods in a working system methods in a working system
implemented and prototype systemsimplemented and prototype systems
Better theoriesBetter theories
Design research outputs
Design research outputs
emergent theory about
emergent theory about
embedded phenomena embedded phenomena knowledge as knowledge as operational principles operational principles
artifact as situated implementation
artifact as situated implementation
abstraction abstraction abstraction abstraction abstraction abstraction constructs constructs better theories better theories models models instatiations instatiations methods methods constructs constructs models models methods methods constructs constructs better theories better theories [Purao , 2002] [Purao , 2002]
Examples
Examples
Open up a new areaOpen up a new area
Provide a unifying frameworkProvide a unifying framework
Resolve a long-standing questionResolve a long-standing question
Thoroughly explore an areaThoroughly explore an area
Contradict existing knowledgeContradict existing knowledge
Experimentally validate a theoryExperimentally validate a theory
Produce an ambitious systemProduce an ambitious system
Provide empirical dataProvide empirical data
Derive superior algorithmsDerive superior algorithms
Develop new methodologyDevelop new methodology
Develop a new toolDevelop a new tool
Artifact structure
Artifact structure
Structure of the artifact
Structure of the artifact
the information space the artifact spansthe information space the artifact spans
basis for deducing all required inbasis for deducing all required informationformation
about the artifact about the artifact
determines the configurationaldetermines the configurational
characteristics necessary to enable the characteristics necessary to enable the evaluation of the artifact
Evaluation criteria
Evaluation criteria
Evaluation criteria
Evaluation criteria
the dimensions of the information spacethe dimensions of the information space
which are relevant for determining the utility which are relevant for determining the utility of the artifact
of the artifact
Evaluation approach
Evaluation approach
Evaluation approach
Evaluation approach
the procedure how to practically test an artifactthe procedure how to practically test an artifact
defines all roles concerned with thedefines all roles concerned with the
assessment and the way of handling the assessment and the way of handling the evaluation
evaluation
result is a decision whether or not the artifactresult is a decision whether or not the artifact
meets the evaluation criteria based on the meets the evaluation criteria based on the available information.
Evaluation approach (2)
Evaluation approach (2)
Quantative evaluation:
Quantative evaluation:
originally developed in the natural sciencesoriginally developed in the natural sciences
to study natural phenomena to study natural phenomena
approaches:approaches:
survey methodssurvey methods
laboratory experimentslaboratory experiments
formal methods (e.g. econometrics)formal methods (e.g. econometrics)
numerical numerical methods methods (e.g. (e.g. mathematicalmathematical
modeling) modeling)
Evaluation approach (3)
Evaluation approach (3)
Qualitative evaluation:Qualitative evaluation:
developed in the social sciences to enabledeveloped in the social sciences to enable
researchers to study social and cultural phenomena researchers to study social and cultural phenomena
approaches:approaches:
action researchaction research
case study researchcase study research
ethnographyethnography
grounded theorygrounded theory
qualitative data sources:qualitative data sources:
observation and participant observation (fieldwork)observation and participant observation (fieldwork)
interviews and questionnairesinterviews and questionnaires
documents and textsdocuments and texts
Constructs
Constructs
S
Sttrruuccttuurree EEvvaalluuaattiioon n ccrriitteerriiaa EEvvaalluuaattiioon n aapppprrooaacchh meta-model of the
meta-model of the vocabulary
vocabulary
construct deficitconstruct deficit
construct overloadconstruct overload
construct redundancyconstruct redundancy
construct excessconstruct excess
ontological analysis ontological analysis
Methods
Methods
S
Sttrruuccttuurree EvEvaalluuaattiioon n ccrriitteerriiaa EEvvaalluuaattiioon n aapppprrooaacchh
process-based metaprocess-based meta
model model
intended applicationsintended applications
conditions conditions of of applicabilityapplicability
products and results of theproducts and results of the
method application method application
reference to constructsreference to constructs
appropriatenessappropriateness completenesscompleteness consistencyconsistency
laboratory researchlaboratory research
field inquiriesfield inquiries
surveyssurveys
case studiescase studies
action researchaction research
practice descriptionspractice descriptions
Models
Models
S
Sttrruuccttuurree EEvvaalluuaattiioon n ccrriitteerriiaa EEvvaalluuaattiioon n aapppprrooaacchh
domaindomain
scope, purposescope, purpose
syntax and semanticssyntax and semantics
terminologyterminology
intended applicationintended application
correctenesscorrecteness completenesscompleteness clarityclarity flexibilityflexibility simplicitysimplicity applicabilityapplicability implementabilityimplementability
syntactical validationsyntactical validation
integrity checkingintegrity checking
sampling using selectivesampling using selective
matching of data to actual matching of data to actual external phenomena or external phenomena or trusted surrogate
trusted surrogate
integration testsintegration tests
risk and cost analysisrisk and cost analysis
Instantiations
Instantiations
S
Sttrruuccttuurree EEvvaalluuaattiioon n ccrriitteerriiaa EEvvaalluuaattiioon n aapppprrooaacchh
executable implementation inexecutable implementation in a programming language
a programming language
reference to a design reference to a design modelmodel
reference to a reference to a requiremenrequirementt specification
specification
reference to thereference to the documentation documentation
reference to qualityreference to quality management documents management documents
reference to configurationreference to configuration management documents management documents
reference to projectreference to project
management documentsmanagement documents
functionalityfunctionality usabilityusability reliabilityreliability performanceperformance supportabilitysupportability
code inspectioncode inspection
testingtesting
code analysiscode analysis
Conclusion
Conclusion
Good research results require a careful
Good research results require a careful
design of the research methodology and
design of the research methodology and
considerable evaluation efforts
Research Concepts
Research Concepts
Agenda
Agenda
Research BasicsResearch Basics
What research is and is notWhat research is and is not
Where research comes fromWhere research comes from
Research deliverablesResearch deliverables
MethodologiesMethodologies
Research processResearch process
Quantitative versus qualitative researchQuantitative versus qualitative research
Research Basics
Research Basics
What research is and isn’tWhat research is and isn’t
Research characteristicsResearch characteristics
Research projects and pitfallsResearch projects and pitfalls
Sources of research projectsSources of research projects
Elements of research proposalsElements of research proposals
What Research Is Not
What Research Is Not
Research isn’t information gathering:Research isn’t information gathering:
Gathering information from resources such books or Gathering information from resources such books or
magazines isn’t research. magazines isn’t research.
No contribution to new knowledge.No contribution to new knowledge.
Research isn’t the transportation of Research isn’t the transportation of facts:facts:
Merely transporting facts from one resource toMerely transporting facts from one resource to
another doesn’t constitute research. another doesn’t constitute research.
No contribution to new knowledge although tNo contribution to new knowledge although this mighthis might
make existing knowledge more accessible. make existing knowledge more accessible.
What Research Is
What Research Is
Research is:Research is:
“…the systematic process of collecting and “…the systematic process of collecting and analyzing information (data) in order to increase analyzing information (data) in order to increase our understanding of the phenomenon about our understanding of the phenomenon about which we are concerned or interested.”
Research Characteristics
Research Characteristics
1.
1. Originates with a question or problem.Originates with a question or problem.
2.
2. Requires clear articulation of a goal.Requires clear articulation of a goal.
3.
3. Follows a specific plan or procedure.Follows a specific plan or procedure.
4.
4. Often divides main problem into subproblems.Often divides main problem into subproblems.
5.
5. Guided by specific problem, question, or hypothesis.Guided by specific problem, question, or hypothesis.
6.
6. Accepts certain critical assumptions.Accepts certain critical assumptions.
7.
7. Requires collection and interpretation of data.Requires collection and interpretation of data.
8.
Research Projects
Research Projects
Research begins with a problem.Research begins with a problem.
This problem need not be This problem need not be Earth-shaking.Earth-shaking.
Identifying this problem can actually be theIdentifying this problem can actually be the hardest part of research.
hardest part of research.
In general, good research projects should:In general, good research projects should:
Address an important question.Address an important question.
Research Project Pitfalls
Research Project Pitfalls
The following kinds of projects usually don’tThe following kinds of projects usually don’t make for good research:
make for good research:
Self-enlightenment.Self-enlightenment.
Comparing data sets.Comparing data sets.
Correlating data sets.Correlating data sets.
High-Quality Research
High-Quality Research
(1 of 2)
(1 of 2)
Good research requires:Good research requires:
The scope and limitations of the work to be clearlyThe scope and limitations of the work to be clearly
defined. defined.
The process to be clearly explained so that it The process to be clearly explained so that it can becan be
reproduced and verified by other researchers. reproduced and verified by other researchers.
A thoroughly planned design that is as objective asA thoroughly planned design that is as objective as
possible. possible.
High-Quality Research
High-Quality Research
(2 of 2)
(2 of 2)
Good research requires:Good research requires:
Highly ethical standards be applied.Highly ethical standards be applied.
All limitations be documented.All limitations be documented.
Data be adequately analyzed and Data be adequately analyzed and explained.explained.
All findings be presented unambiguously and allAll findings be presented unambiguously and all
conclusions be justified by sufficient evidence. conclusions be justified by sufficient evidence.
Sources of Research Problems
Sources of Research Problems
Observation.Observation.
Literature reviews.Literature reviews.
Professional conferences.Professional conferences.
Stating the Research Problem
Stating the Research Problem
Once you’ve identified a research problem:Once you’ve identified a research problem:
State that problem clearly and State that problem clearly and completely.completely.
Determine the feasibility of the research.Determine the feasibility of the research.
Identify subproblems:Identify subproblems:
Completely researchable units.Completely researchable units.
Small in number.Small in number.
Add up to the total problem.Add up to the total problem.
Hypotheses
Hypotheses
Hypotheses are tentative, intelligent guessesHypotheses are tentative, intelligent guesses as to the solution of the problem.
as to the solution of the problem.
There is often a 1-1 correspondence between aThere is often a 1-1 correspondence between a
subproblem and a hypothesis. subproblem and a hypothesis.
Hypotheses can direct later research activities sinceHypotheses can direct later research activities since
they can help determine the nature of the research they can help determine the nature of the research and methods applied.
Delimitations
Delimitations
All research has limitations and thus certainAll research has limitations and thus certain work that
work that willwill notnot be performed.be performed.
The work that will not be undertaken isThe work that will not be undertaken is described as the
Definitions
Definitions
Define each technical term as it is used inDefine each technical term as it is used in relation to your research project.
relation to your research project.
This helps remove significant ambiguity from theThis helps remove significant ambiguity from the
research itself by ensuring that reviewers, while they research itself by ensuring that reviewers, while they may not agree with your definitions, at least know may not agree with your definitions, at least know what you’re talking about.
Assumptions
Assumptions
Assumptions are those things that theAssumptions are those things that the researcher is taking for granted.
researcher is taking for granted.
For example: a given test instrument accurately andFor example: a given test instrument accurately and
consistently measures the phenomenon in question. consistently measures the phenomenon in question.
As a general rule you’re As a general rule you’re better off documentingbetter off documenting an assumption than ignoring it.
an assumption than ignoring it.
Overlooked assumptions provide a prime source of Overlooked assumptions provide a prime source of
debate about a research project’s results. debate about a research project’s results.
Importance of the Study
Importance of the Study
Many research problems have a kind of Many research problems have a kind of
theoretical feel about them. Such projects o
theoretical feel about them. Such projects oftenften need to be justified:
need to be justified:
What is the research project’s practical value?What is the research project’s practical value?
Without this justification, it will prove difficult toWithout this justification, it will prove difficult to convince others that the problem in question is convince others that the problem in question is worth study.
Research Proposals
Research Proposals
Research proposals are documents thatResearch proposals are documents that describe the intended research including: describe the intended research including:
Problem and subproblems.Problem and subproblems.
Hypotheses.Hypotheses. Delimitations.Delimitations. Definitions.Definitions. Assumptions.Assumptions. Importance.Importance.
Literature Review
Literature Review
A literature review is a necessity.A literature review is a necessity.
Without this step, you won’t know if yWithout this step, you won’t know if your problem hasour problem has
been solved or what related research is already been solved or what related research is already underway.
underway.
When performing the review:When performing the review:
Start searching professional journals.Start searching professional journals.
Begin with the most recent articles you can find.Begin with the most recent articles you can find.
Keep track of relevant articles in a Keep track of relevant articles in a bibliography.bibliography.
Don’t be discouraged if work on the topic is Don’t be discouraged if work on the topic is alreadyalready
underway. underway.
Literature Review Pitfalls
Literature Review Pitfalls
(1 of 2)
(1 of 2)
Be very careful to check your sources Be very careful to check your sources whenwhen doing your literature review.
doing your literature review.
Many trade magazines are not peer reviewed.Many trade magazines are not peer reviewed.
Professional conferences and journals often haveProfessional conferences and journals often have
each article reviewed by multiple people before it is each article reviewed by multiple people before it is even recommended for publication.
even recommended for publication.
The IEEE and ACM digital libraries are good placesThe IEEE and ACM digital libraries are good places
to start looking for legitimate research. to start looking for legitimate research.
Literature Review Pitfalls
Literature Review Pitfalls
(2 of 2)
(2 of 2)
The Internet can be a good source of The Internet can be a good source of
information. It is also full of pseudo-science and information. It is also full of pseudo-science and poor research.
poor research.
Make sure you verify the claims of anyMake sure you verify the claims of any documentation that has not been peer documentation that has not been peer reviewed by other professionals in the reviewed by other professionals in the computing industry.
Processes & Methodologies
Processes & Methodologies
Research Process.Research Process.
Common Methodologies.Common Methodologies.
Research Process
Research Process
Research is an extremely cyclic process.Research is an extremely cyclic process.
Later stages might necessitate a review of earlier Later stages might necessitate a review of earlier
work. work.
This isn’t a weakness of the process but is This isn’t a weakness of the process but is partpart of the built-in error correction machinery.
of the built-in error correction machinery.
Because of the cyclic nature of research, it canBecause of the cyclic nature of research, it can be difficult to determine where to start and
be difficult to determine where to start and when to stop.
Step 1: A Question Is Raised
Step 1: A Question Is Raised
A question occurs to or is posed to A question occurs to or is posed to thethe
researcher for which that researcher has no researcher for which that researcher has no answer.
answer.
This doesn’t mean that someone else This doesn’t mean that someone else doesn’t alreadydoesn’t already
have an answer. have an answer.
The question needs to be converted to anThe question needs to be converted to an appropriate problem statement like that appropriate problem statement like that documented in a research proposal.
Step 2: Suggest Hypotheses
Step 2: Suggest Hypotheses
The researcher generates intermediateThe researcher generates intermediate hypotheses to describe a solution to the hypotheses to describe a solution to the problem.
problem.
This is at best a tThis is at best a temporary solution since there is asemporary solution since there is as
yet no evidence to support either the acceptance or yet no evidence to support either the acceptance or rejection of these hypotheses.
Step 3: Literature Review
Step 3: Literature Review
The available literature is reviewed toThe available literature is reviewed to
determine if there is already a solution to the determine if there is already a solution to the problem.
problem.
Existing solutions do not always explain newExisting solutions do not always explain new
observations. observations.
The existing solution might require some revision The existing solution might require some revision or or
even be discarded. even be discarded.
Step 4: Literature Evaluation
Step 4: Literature Evaluation
It’s possible that the literature review hasIt’s possible that the literature review has yielded a solution to the proposed problem. yielded a solution to the proposed problem.
This means that you haven’t really done research.This means that you haven’t really done research.
On the other hand, if the literature review turnsOn the other hand, if the literature review turns up nothing, then additional research activities up nothing, then additional research activities are justified.
Step 5: Acquire Data
Step 5: Acquire Data
The researcher now begins to gather dataThe researcher now begins to gather data relating to the research problem.
relating to the research problem.
The means of data acquisition will often changeThe means of data acquisition will often change
based on the type of the research problem. based on the type of the research problem.
This might entail only data gathering, but it could alsoThis might entail only data gathering, but it could also
require the creation of
require the creation of new measurementnew measurement instruments.
Step 6: Data Analysis
Step 6: Data Analysis
The data that were gathered in the previousThe data that were gathered in the previous step are analyzed as a first step in ascer
step are analyzed as a first step in ascertainingtaining their meaning.
their meaning.
As before, the analysis of the data does notAs before, the analysis of the data does not constitute research.
constitute research.
Step 7: Data Interpretation
Step 7: Data Interpretation
The researcher interprets the newly analyzedThe researcher interprets the newly analyzed data and suggests a conclusion.
data and suggests a conclusion.
This can be difficult.This can be difficult.
Keep in mind that data analysis that suggests aKeep in mind that data analysis that suggests a
correlation between two variables can’t
correlation between two variables can’t automaticallyautomatically be interpreted as suggesting causality between those be interpreted as suggesting causality between those variables.
Step 8: Hypothesis Support
Step 8: Hypothesis Support
The data will either support the hypotheses or The data will either support the hypotheses or they won’t.
they won’t.
This may lead the researcher to cycle back to This may lead the researcher to cycle back to anan
earlier step in the process and begin again with a earlier step in the process and begin again with a new hypothesis.
new hypothesis.
This is one of the self-correcting mechanismsThis is one of the self-correcting mechanisms
associated with the scientific method. associated with the scientific method.
Common Methodologies
Common Methodologies
Methodologies are high-level approaches toMethodologies are high-level approaches to conducting research.
conducting research.
The individual steps within the The individual steps within the methodology mightmethodology might
vary based on the research being performed. vary based on the research being performed.
Two commonly used research methodologies:Two commonly used research methodologies:
Quantitative.Quantitative.
Methodology Comparison
Methodology Comparison
Quantitative Quantitative
Explanation, predictionExplanation, prediction
Test theoriesTest theories
Known variablesKnown variables
Large sampleLarge sample
Standardized instrumentsStandardized instruments
DeductiveDeductive
Qualitative Qualitative
Explanation, descriptionExplanation, description
Build theoriesBuild theories
Unknown variablesUnknown variables
Small sampleSmall sample
Observations, interviewsObservations, interviews
