R
An Approach to the Life Cycle
Evaluation of Complex Systems
MIT Humans and Technology Symposium, January 23, 2006
Donald Cox, Bill Long, Lisa Stevens
Donald Cox, Bill Long, Lisa Stevens
Sterling Wiggins, Cindy Dominguez, Tom Miller
Sterling Wiggins, Cindy Dominguez, Tom Miller
Klein Associates Division, ARA
Klein Associates Division, ARA
donald@decisionmaking.com
Thesis
In order to provide useful, practical,
theoretically grounded feedback on the
development of complex socio-technical
systems, we must bring individual
techniques together to provide an
evolving view of the system as it
develops.
Cognitive(!)
Metrics!!
On Metrics
There are too many simplistic metrics that
don't capture the essence of whatever it is that they are supposed to measure. There are too many uses of simplistic measures that don't even recognize what attributes are supposedly being measured. Starting from a detailed
analysis of the task or attribute under study might lead to more complex, and more
qualitative, metrics, but we believe that it will also lead to more meaningful and therefore more useful data.
Evaluation
{
Measurement
{Scoring
Measurement
{
Observation of a quantitative
attribute
{
Requires that the attribute be shown
to have the appropriate structure
{
“Quantitative structure is but one
(important) kind amongst many and
it holds no franchise over scientific
method in its entirety.” (Michell,
1999)
Scoring
{
Quantification for the purpose of
comparison
{
Examples: EPA MPG testing, school
grades
{
Useful given a substantial empirical
base
Assessment
“Assessment, however, is concerned
with determining the significance,
importance, or value of an event and
refers to the procedures used to obtain
information and form value
judgments…”
What Now?
“To be effective, however, an
evaluation program must
match the dynamics of the
system to which it is applied.”
Safety Case
We define a safety case as:
“A documented body of evidence that provides a convincing and valid argument that a system is adequately safe for a given application in a given environment”
To implement a safety case we need to
◦ make an explicit set of claims about the system
◦ produce the supporting evidence
◦ provide a set of safety arguments that link the claims to the evidence
◦ make clear the assumptions and judgments underlying the arguments
◦ allow different viewpoints and levels of detail
Cognitive Performance Case
We define a cognitive performance case as:
“A documented body of evidence that provides a convincing and valid argument that a system enable adequate cognitive performance for a given application in a given environment”
To implement a cognitive performance case we need to:
◦ make an explicit set of claims about the system
◦ produce the supporting evidence
◦ provide a set of cognitive performance arguments that link the claims to the evidence
◦ make clear the assumptions and judgements underlying the arguments
Constructing the Case
Cognitive Evaluation Cognitive Assessment Expert Review Macrocognitive Walkthrough Observation of Use Prototype InterviewAssessment Techniques
Observation of Use High Prototype Interview Resource Demand Macrocognitive Walkthrough Expert Review Low Concept Deployment System MaturityCognitive Assessment
Techniques
Expert Review
Value:
Early and agile investment of expertisePurpose:
Map out where to work in the short- and long-termActivity:
Combine individualworking-through of system implications and unknowns
Macrocognitive Walkthrough
Value:
Theoretical description of using the systemPurpose:
Find issues by systematic examination of the systemActivity:
Task- and team-based working-through of how someone will construct performanceOutput:
Detailed qualitative model of human-system interactionPrototype Interview
Domain-based account of what
does or doesn’t work in the design
Output:
Conversation with a person using the system to accomplish their work
Activity:
Engage users to uncover bad
design assumptions and remaining tacit knowledge/practice
Purpose:
Checking our assumptions with the real world of work
Observation of Use
Value:
Working out what actually goes on in creating performancePurpose:
See actual performance in detailActivity:
Observation and reconstruction of use in a natural settingOutput:
Understanding of how work is accomplished with the systemCognitive Assessment Guidance
Indicators
Distilling Expertise
Cognitive Assessment Guidance NDM/CSE LiteratureCSE Practitioners
Klein Associates Gary Klein Recognition-Primed Decisions SA Technologies Mica Endsley SA-Oriented Design Ohio State David WoodsLaws that Govern Cognitive Work
Institute for Human and Machine Cognition
Robert Hoffman
Human Centered Computing
University of Toronto
Kim Vicente
Cognitive Work Analysis
There are similarities around how CSE practitioners
describe and identify systems that support/hinder cognitive functions
Guidance Indicators
{ Option Workability { Cue Prominence { Direct Comprehension { Fine Distinctions { Transparency { What-If Capabilities { Enabling Anticipation { Historic Information { Direct Involvement { Situation Assessment { Directability { Flexibility in Procedures { Adjustable SettingsOrganizational Change
{
Why do more?
CPC Summary
{
Need for evaluation
{Structured argument
{
Assessment techniques
{
