Rating Scales A Fundamental Analytic Method

(1)

February 2012

Rating Scales

(2)

2

This presentation provides an introduction to using Rating Scales for

market insights analysts with limited experience using this approach

Topics

• Why rating scales work

• When to use a rating scales

• How to create a rating scale

• Which type of scale to use

(3)

Have you ever been in a meeting when people had different points

of view and had a hard time reaching agreement?

No no no, you don’t

understand, our top

priority should be

city zoos!

This guy

belongs in a

(4)

4

No need to suffer endless debates ... rating scales simplify complex

topics, bring objectivity to potentially emotional topics, are efficient to

use, and lead to action



Simplify complex topics

– Reduce multiple complex considerations into

summary measures.



Objective, less contentious

– Removes ambiguity making the rating factors

explicit and easier to reach agreement on.



Efficient

– actual rating steps take less time and validation is easier with subject

matter experts and with stakeholders.



Actionable

– action recommendations are “designed in” from the start.

Advantages of Defined Rating Scales

(5)

5

Many of us had our first experience of rating scales when we began

to receive grades in school

(6)

6

School grades combine scores from several different rating factors

into a single score that can drive specific actions

Grade

Components

• Class

participation

20%

• Homework

30%

• Tests and

Quizzes 50%

Action from

Grades

• Advanced

class

placement

• Tutoring

• Admission to

college

Expert Scoring of Class

Achievement

(7)

7

Scoring models can also be created by crowd-sourcing, which has

become typical for restaurants, and other entertainment

Crowd-sourced

Restaurant Scoring

Action from

Score

• Diner:

Select

restaurants to

try out

• Owner:

Improve items

with low

scores

Rating

Components

• Food

• Service

• Ambiance

• Price

(8)

For movies, we have ratings for the appropriate age range based on

several components and used for “filtering”

8

Expert Movie Rating

for Age/Maturity

Rating

Components

• Sexuality

• Language

• Violence

• Drug use

Action from

Rating

• Control who

is

permitted

to attend a

movie

• Select a

movie I

might

prefer

(9)

Bestseller and “Top 40” lists are a special type of rating based

primarily on categorization and counting to produced ranked lists

1. TAKEN, by Robert Crais

2. PRIVATE: #1 SUSPECT, by James Patterson and Maxine Paetro

3. DEATH COMES TO PEMBERLEY, by P. D. James

4. THE GIRL WHO KICKED THE HORNET’S NEST, by Stieg Larsson

5. 11/22/63, by Stephen King

1. AMERITOPIA, by Mark R. Levin

2. AMERICAN SNIPER, by Chris Kyle with Scott McEwen and Jim DeFelice

3. STEVE JOBS, by Walter Isaacson

4. QUIET, by Susan Cain

5. KILLING LINCOLN, by Bill O'Reilly and Martin Dugard

Source: Feb. 12 -- http://www.nytimes.com/best-sellers-books/overview.html

NYT Bestsellers

2 of 23 categories

Rating

Elements

• Categorize

each book

• Count sales

for the week

Action from

Rating

• Reader:

What books

should I

evaluate for

reading?

• Bookseller:

What books

should I

carry in my

store?

(10)

Detailed, granular ratings can help identify specific issues that need

attention

Gartner Assessment of Outsourcing Locations 2008

Source: http://www.zdnet.com/blog/btl/gartner-rates-offshore-outsourcing-hot-spots/10388

Action from

Rating

• Select

potential

outsourcing

locations

• Develop

amelioration

plans

(11)

11

And, combining two ratings together can provide a strategic view as

in the Gartner magic quadrant

Action from

Rating

• Customer:

Select

potential

vendors

• Vendor:

Focus on

key

competitors

http://www.gartner.com/DisplayDocument?doc_cd=131166#h-N65811

Rating Elements

• Completeness of vision • Market understanding • Marketing strategy • Sales strategy • Offering strategy • Business model • Industry strategy • Innovation • Geographic strategy • Ability to execute • Product/service • Overall viability • Sales Execution/Pricing • Market responsiveness and track record • Marketing execution • Customer experience • Operations

2010 Gartner Magic Quadrant for

Application Performance Monitoring

(12)

12

Or, as in this IDC example, two categorizations can be combined to provide

a strategic view

Engineering

Oriented

Technology

Oriented

Asset

Oriented

Brand

Oriented

Manufacturing Value Chain Segmentation Focus

Supply

Complexity

Demand

Complexity

Low

High

Low

High

(13)

While there are many different types of rating scales, most projects will use

one of these three core methods

Scoring (grades)

• Which is BEST?

• Use when “best” can be defined

• Extremely versatile

• Single or multi-dimensional

• Decimal accuracy to Harvey balls

Three Types of Rating Scales

Counting (bestsellers)

• Which is MOST?

• Use when “most” is a good proxy for

importance; e.g., what is the most

frequent enhancement request?

Grouping (movie rating)

• What GROUP is it in?

• Use when complex logic is needed

• Often used for filtering

• Often used for segmentation and

sales targeting

(14)

Let’s look at one example of using a scoring system

Scoring (grades)

• Which is BEST?

• Use when “best” can be defined

• Extremely versatile

• Single or multi-dimensional

• Decimal accuracy to Harvey balls

Three Types of Rating Scales

Counting (bestsellers)

• Which is MOST?

• Use when “most” is a good proxy for

importance

• What’s frequently requested?

• How many units shipped?

Grouping (movie rating)

• What GROUP is it in?

• Use when complex logic is needed

• Often used for filtering

• Often used for segmentation and

sales targeting

(15)

To develop a scoring rating scale, define rating categories, create a scale

for each category, rate using the scale, validate, and present

1. Define rating categories:

–

Make categories actionable.

2. Create a defined scale for each

category:

–

Divide into sub-elements that can be

rated.

–

Define scales to create objectivity.

3. Rate using the scale and validate

with SME’s:

–

Fact-based sources and SME

validation create confidence.

4. Present and recommend action:

–

Ratings should map to actions.

Steps for

(16)

For example, for a project designed to recommend improvement areas for

CIO’s, 10 capability areas were defined that CIO’s could act on

Capability Area (action areas)

a) Business and IT Relationship

b) Client Devices and User Support

c) IT Appl. And Service Management

d) IT Project Mgmt. & Execution

e) IT Fin. Mgmt. & Procurement

f) IT Organization and Skills

g) Server and Storage Management

h) Network

i) Service Delivery and Support

j) Site and Facility

(17)

For each of the capability areas, we divide them into sub-elements that can

be clearly defined and rated – we leveraged ITIL models to create these

Capability Area (action areas)

a) Business and IT Relationship

b) Client Devices and User Support

c) IT Appl. And Service Management

d) IT Project Mgmt. & Execution

e) IT Fin. Mgmt. & Procurement

f) IT Organization and Skills

g) Server and Storage Management

h) Network

i) Service Delivery and Support

j) Site and Facility

Now +2yr Bus IT

a) Business and IT Relationship

0.0 0.0 0.0 0.0

IT Mission - as understood by the business 0.0 0.0 IT Communication with Business Areas 0.0 0.0 IT Investment Prioritization 0.0 0.0

Metrics and Measurements 0.0 0.0

(18)

The defined rating scale provides clear definitions with an objective

meaning, as well as implied improvement guidance

Now +2yr Bus IT ₁ ₂ ₃

a) Business and IT Relationship 0.0 0.0 0.0 0.0

IT Mission - as understood by the business 0.0 0.0 Manage technology -- technical focus. Provide applications to support business areas -- individual application focus.

Help business areas -- leveraging technology for process excellence.

IT Communication with Business Areas

0.0 0.0 Ad-hoc, based mainly on requests for support or complaints.

Assigned support for business areas with an assessment of needs by dept. Regular communications focused on improving impact of IT on business results. IT Investment Prioritization

0.0 0.0 Priority of projects based on opinions more than analysis, typically with an annual list of projects only for budgeting.

Consistent framework for identifying, documenting, and prioritizing projects with sign-off by a business steering committee.

Significant IT projects tied to business

initiatives with business owners; or have

measurable

infrastructure goals.

Metrics and Measurements

0.0 0.0 Metrics limited to those created by individual products and

applications, with limited or no reporting to the business.

Metrics are summarized into a monthly or

quarterly reports, reveiwed by

management, that reflect technology-centric SLA's.

Monthly or quarterly review of IT-related metrics specific to support of major

business processes and departments; example: help desk calls by department with resolution.

1

3

5

(19)

Business and IT pain point scales were also defined, but as a single

scale used across all capabilities

Business Pain Points

Minimal Issues

Minor Concerns

Significant Issue

Critical Issue

0 1 2 3

Problems in this area are within the range of normal business operations. No significant concerns.

There are some problems; however, IT management has plans to address the issues, which have been communicated.

Problems in this area have some business impacts and require the attention of non-IT management, and improvement appears slow.

Problems in this area are significantly affecting business operations, and significant changes are needed within the next 18 months.

IT Pain Points

Minimal Issues

Minor Concerns

Significant Issue

Critical Issue

0 1 2 3

Problems in this area are within the range of normal business operations. No significant concerns.

There are some problems; however impacts are limited to IT internal operations and IT

management has a plan to address the issues.

Inefficiencies or problems in this area require an excessive amount of IT management or IT staff time, which affects IT productivity and effectiveness.

Problems in this area are causing or may soon cause dissatisfaction with IT, or are absorbing far too much management attention. Issues need to be addressed within the next 18 months.

(20)

Ratings were done collaboratively with customer teams in

face-to-face workshops, so the results were easily accepted

(21)

Current Capabilty Level and Gap to +2yr Target

0.0 0.5 1.0 1.5 2.0 2.5 3.0 3.5 4.0 4.5 5.0

a) Business & IT Relationship

b) Client devices and User Support

c) IT Applications and Services Management

d) IT Project Management and Execution

e) IT Financial Management and Procurement

f) IT Organization and Skills

g) Server and Storage Management

h) Network

i) Service Delivery and Support

j) Site and Facility

Cap. Level Now Cap. Level Gap

Customers and IBM’ers left the meetings with a shared agenda for areas

needing improvement

Average of Current Capability Levels

Source: IBM Market Insights, onsite workshop, September 2nd, 2009

Note: pain

scores were also incorporated into improvement prioritization

(22)

While there are many different types of rating scales, most projects will use

one of these three core methods

Scoring (grades)

• Which is BEST?

• Use when “best” can be defined

• Extremely versatile

• Single or multi-dimensional

• Decimal accuracy to Harvey balls

Three Types of Rating Scales

Counting (bestsellers)

• Which is MOST?

• Use when “most” is a good proxy for

importance

• What’s frequently requested?

• How many units shipped?

Grouping (movie rating)

• What GROUP is it in?

• Use when complex logic is needed

• Often used for filtering

• Often used for segmentation and

sales targeting

(23)

For a counting-based rating scale, identify major categories, define

sources, categorize findings, refine categories, present and

recommend actions

1. Identify major categories

–

Make categories actionable.

2. Define sources and collection

protocols

–

The data pool should be meaningful.

3. Categorize, subcategorize, and

recategorize until the categories

seem meaningful and clear

–

This is a creative process.

4. Present and recommend action:

–

Ratings should map to actions.

Steps for

(24)

For example, for a project designed to identify areas to improve in IBM’s

cloud messaging, we identified seven message categories and four

competitors

Message Categories

Average

Service Delivery Attribute

Business benefits

Audience

Partnership association

Workloads

Industry

Negative/Critical Messaging

Totals

100%

Significant high level of messaging Significant low level of messaging

(25)

25 Feb 2012 Rating Scales Dec 2009

To capture messages that a typical cloud buyer would see, we used

a standard protocol for gathering information from six sources

Messaging Sources

Standard Collection Protocol

Analyst Reports

Reports identified using simple search using competitor and cloud _{reference. Top 4 rated reports were read for each competitor and IBM.}

Whitepapers/Brochures

Total of 3 Whitepapers and Brochures per competitor and IBM. Generally

found on websites or by conducting a web search.

Industry Websites

Three industry websites were targeted: CIO.com, eWeek.com and Techtarget.com. 3 articles per competitor and IBM were read and messages parsed.

Factiva Search (News)

Conducted a simple search using competitor name and cloud computing

reference. 3 most recent news reports/press releases were evaluated.

Company Websites

Collected messaging from home page and pages directly linked from the

home page (1 click page depth).

Google Search

Simple search conducted using competitor name and cloud computing. Followed any organic links from first two pages that were not covered by an additional source already mentioned above.

In total we captured messages from a 105 individual sources for the four competitors and IBM. Data was

collected during two weeks: November 9 to 22, 2009.

(26)

26

Analyzing the actual messaging around service delivery attributes, we

developed sub-categories and counted message volume

Dec 2009

Service Delivery Attributes:

No. of Messages by Top Five Sub-Categories

# of m es s ag es 37 45 27 18 18 18 19 13 6 6 12 15 24 19 17 22 22 11 5 7 32 12 14 19 9 0 20 40 60 80 100 120 140

Amazon Google IBM Microsoft Salesforce.com

Service Delivery Attribute

sub-category definitions

Ease of use

Offering that is easy to use and builds on existing skills, frameworks and languages

Scalability

Ability to scale applications up and down to manage demand shifts

Security

Mention of a secure environment or specific security plans/policies

Speed of implementation

Speed of provisioning and deployment

Service Quality

Service availability, reliability and

dependability with features such as fault tolerance

(27)

In this case we also evaluated the quality of messaging to highlight

areas for greater clarity

Dec 2009

Competitors

Amazon

“simple web service interface allows you to obtain and configure capacity with minimal friction”

Google

“End users can use the familiar Microsoft Outlook interface for email, contacts and calendar as they transition to Gmail and Google Calendar”

Microsoft

“use their existing skills to develop cloud

applications”,” familiar development experience”

Salesforce.com

“Force.com automatically generates a rich user interface that you can customize with our drag-and-drop page layout editor”

IBM

IBM

“could simplify the often arduous process” – CloudBurst, Ovum Report

“simple-to-use, self-service test platform” – Smart Business Development and Test, IBM Website “create a dynamic IT infrastructure that is easier and less expensive to manage, upgrade and run” – Blue Cloud, IBM Website

“simple to invoke” – CloudBurst, IBM Whitepaper

Source: Cloud Messaging Project Data, November 2009

Microsoft and Google provide more specific information about how ease of use

is achieved than IBM. This could be an area for greater clarity in our

messaging.

(28)

We leveraged the quantitative ratings and quality assessments to provide

specific recommendations to improve IBM’s messaging

Dec 2009

Source: MI Analysis

1 Security certifications

- Competitors are providing details regarding existing certifications as well as efforts to obtain new certifications

2 Customer references/case studies

- Customer video references and specific case studies regarding service usage

3 Third party sponsored whitepapers

- ROI sponsored whitepapers lend analyst credibility to competitor marketing messages

4 Partnership associations

- New emerging competitors such as Salesforce.com are establishing partnerships with established industry brand names and giving credibility to their solutions

5 Quantification of benefits

- Detailed and specific information provided for messaging sub-categories

(29)

While there are many different types of rating scales, most projects will use

one of these three core methods

Scoring (grades)

• Which is BEST?

• Use when “best” can be defined

• Extremely versatile

• Single or multi-dimensional

• Decimal accuracy to Harvey balls

Three Types of Rating Scales

Counting (bestsellers)

• Which is MOST?

• Use when “most” is a good proxy for

importance

• What’s frequently requested?

• How many units shipped?

Grouping (movie rating)

• What GROUP is it in?

• Use when complex logic is needed

• Often used for filtering

• Often used for segmentation and

sales targeting

(30)

To develop a grouping or segmentation rating, define the groups and the

logic of group membership, categorize based on data, and recommend

actions based on group membership

1. Define groups

–

Make groups actionable

2. Define logic / rules for

membership in each group

–

Can be complex and even statistically

based, but must be explainable

3. Categorize into groups using the

logic / rules and validate with

SME’s

–

Ensure that SME’s think your

categorization makes sense

4. Present and recommend action:

–

Groups should map to actions

Steps for

(31)

31

The paramount step was to prioritize the accounts in XXX and XXX to

identify the customers that would be best candidates for ibm.com

Attributes

Category Examples

Revenue System x revenue in 08, 09, 10.

Industry Computer services; Insurances; etc. Employees Number of employees Won revenue STG total won revenue

(ssc=07, 08) in 08’-10’ Opp. Owning

Channels

Business Partner; Face-to-Face; ibm.com MAP Industry

Group

Financial services sector; Distribution sector; etc.

Key Predictive

Attributes of those

that ‘Look-Like’ best

customers

Accounts receive a score which indicates the extent to

which they are suitable to be destacked to ibm.com.

XXX and XXX space (Both Destacked and Non-destacked accounts)

Identify Key Drivers

Develop Inputs

Find Patterns and

Relationships

Relative importance - an example

*Source: MAP and EIW

MAP Ind. Goup 95% IBM.CO M Won Rev. Share 5%

(32)