Measuring impact revisited

(1)

E.V . E.V . O RMA T ION O RMA T ION

Measuring impact revisited

W ERKINF O W ERKINF O ÜR NETZ W ÜR NETZ W

Frank Scholze

ITIA T IVE F ITIA T IVE F

Stuttgart University Library

5th Workshop on Innovations in Scholarly Communication

TSCHE IN I TSCHE IN I

p

y

CERN, Geneva, 18.4.2007

DEUDEU

(2)

Outline

E.V . E.V .

1. Scope / definitions

O RMA T ION O RMA T ION

1. Scope / definitions

2. Issues

3 Ongoing o k / p ojects

W ERKINF O W ERKINF

O

3. Ongoing work / projects

ÜR NETZ W ÜR NETZ W ITIA T IVE F ITIA T IVE F TSCHE IN I TSCHE IN I DEUDEU

(3)

Impact

E.V . E.V . O RMA T ION O RMA T

ION

“Impact is any change resulting from an

activity, project, or organization. It includes

W ERKINF O W ERKINF O

y, p j

,

g

intended as well as unintended effects,

negative as well as positive, and long-term as

ÜR NETZ

W

ÜR NETZ

W

negative as well as positive, and long term as

well as short-term.”

From: Susan Wainwright. Measuring Impact - A Guide to Resources. NCVO, 2002

TSCHE IN

I

TSCHE IN

I

(4)

Measuring I

E.V . E.V .

• What

O RMA T ION O RMA T

ION

– Impact vs. output

• Publications are the quantifiable output of the

research process

ÜR NETZ

W

ÜR NETZ

W

• Electronic publications as a limited but rather

Electronic publications as a limited but rather

well defined sub-field of research output

TSCHE IN

I

TSCHE IN

I

• Which publications represent „e-science“?

(5)

Measuring II

E.V . E.V .

• How

ION

– Choose indicators for publications (citation, usage,

„endorsement“, post-publication peer-review …)

– Collect information

– Analyse / digest the information

ÜR NETZ

W

ÜR NETZ

W

Analyse / digest the information

ITIA T IVE F ITIA T IVE F TSCHE IN I TSCHE IN I DEUDEU

(6)

Indicators and methods

E.V

.

E.V

.

• Web Analytics = Usage

y

g

– Logfile analysis

– Server side scripts (e.g. Linkresolver)

– Page tagging, web bugs, cookies

– Hybrid methods

ÜR NETZ W ÜR NETZ W

• Citation Analysis

E

i

ti

f th f

d

tt

f it ti

– Examination of the frequency and pattern of citations

in publications

– Impact factor (IF)

TSCHE IN

I

TSCHE IN

I

p

( )

(7)

A taxonomy of metrics

E.V

.

E.V

. authors

scholarship evaluation now

O RMA T ION O RMA T ION Google’s PR W ERKINF O W ERKINF O Google s PR Technorati

IF, citation counts

ÜR NETZ W ÜR NETZ W Amazon.com Collaborative filtering Flickr.org Slashdot.org Frequentist “counting” Structural “pattern” ITIA T IVE F ITIA T IVE F g g TSCHE IN I TSCHE IN I users novel methods of resource evaluation

LANL experiments for scholarship evaluation since 1999

DEUDEU _{From: Bollen, Johan and Van de Sompel, Herbert (2005) A framework for assessing the impact of units of}

(8)

Measuring impact:

the elements (schematic)

E.V

.

E.V

.

Services

(ranking, recommender, collection building

the elements (schematic)

Metrics

(structural, quantitative)

and management...)

s

e

Data mining

Metrics (structural, quantitative)

analy

s

Link Resolvers

Logfile analysis

Citation analysis

Compare, aggregate

ect

Link-Resolvers

Collect

A

t

Logfile analysis

Collect

A

t

Citation analysis

Parse

E t

t

coll

Aggregate

Extract

Aggregate

f

(

o

ose

DEUDEU

Basic set of documents

(Journals, repositories,

primary data etc.)

ch

(9)

Outline

E.V . E.V .

1. Scope / definitions

2. Issues

3 Ongoing o k / p ojects

O

3. Ongoing work / projects

(10)

What are the basic items we want to count?

E.V

.

E.V

.

• Granularity (journals, articles, etc.)

• Beyond “publications” (primary data learning

W

ERKINF

O

W

ERKINF

O

• Beyond publications (primary data, learning

objects, etc.)

ÜR NETZ

W

ÜR NETZ

W

• Identifying items (=referent deduplication)

– Metadata based heuristics, persistent identifiers

TSCHE IN

I

TSCHE IN

I

• Identifying users (=agent deduplication)

– User and session identification, pseudonymization

(11)

Indicators and methods

E.V

.

E.V

.

• Log data processing

ION

– Identifying / filtering robots and crawlers

– Proxies, caching

– “click spans”

click spans

– Grouping, isolating and aggregating useful usage patterns

– Comparison and validation to citation data

ÜR NETZ

W

ÜR NETZ

W

• Aggregation and scalability

– Technical issues

Technical issues

• Different architectural frameworks: linking

server-based, other, scalability, pseudonymization

– Social/Policy issues

TSCHE IN

I

TSCHE IN

I

Social/Policy issues

• Networks of trust, sharing usage data

– How is data collated with external sources (publisher data

…)

DEUDEU

(12)

Confidence in data and stats?

E.V . E.V .

• Data validity

ION

– Usage definition, recording and representation, quality

benchmarks, falsification issues

• Fraud and data manipulation

– Standardized and transparent collection and aggregation of

ÜR NETZ

W

ÜR NETZ

W

Standardized and transparent collection and aggregation of

usage data

– Auditing standards and procedures

(13)

Legal issues, metrics and services

E.V

.

E.V

.

• Privacy and other legal issues

y

g

– User and session identification, pseudonymization

– Legal implications of log storage and aggregation

• Metrics and services

ÜR NETZ

W

ÜR NETZ

W

_{– Investigating and tailoring metrics}

– Interfaces with existing bibliometric products

– Definition of end-user services

(14)

Background Issues

E.V

.

E.V

.

• What is the context for talking about impact?

g

p

• Why is promoting enhanced and alternative

metrics important?

– as a licensing issue: consortia and publishers want to

investigate value for money issues

ÜR NETZ

W

ÜR NETZ

W

– as a research policy issue:

• scholarly communication changes rapidly

i

OA

i

• impact in new OA environment

• trend analysis (“in-progress” communication)

• assessment and evaluation of research

TSCHE IN

I

TSCHE IN

I

• assessment and evaluation of research

(15)

Outline

E.V . E.V .

1. Scope / Definitions

2. Issues

3 Ongoing o k / p ojects

O

3. Ongoing work / projects

(16)

Knowledge Exchange Workshop

on Institutional Repositories

E.V . E.V .

on Institutional Repositories

P

ti l d fi iti

f ’

’

Practical definition of ’usage’

Raw format for exchanging usage events

Items to be counted

Lobby COUNTER to add article level stats

W

ERKINF

O

W

ERKINF

O

Lobby COUNTER to add article level stats

Other academic “output types”

Standard reports

Agree on a small set of standard useful statistical reports that

ÜR NETZ

W

ÜR NETZ

W

Agree on a small set of standard useful statistical reports that

repositories should produce

Policies for stats

Compliance with local laws on e.g. privacy

p

g p

y

Enhance SHERPA policy tool

Collection and aggregation

Normalisation

TSCHE IN

I

TSCHE IN

I

Specify issues of aggregation and deduplication for later study

Collation with external sources

Aggregating COUNTER stats at consortium level

I

ti

t SUSHI i t

bilit

ith

it i

d OAI PMH

DEUDEU

Investigate SUSHI interoperability with repositories and OAI-PMH +

OpenURLContextObjects

(17)

Ongoing work

E.V . E.V .

g

• LANL

bX (with CalState ExLibris)

– bX (with CalState, ExLibris)

– MESUR

– …

• UK

ÜR NETZ W ÜR NETZ

W

_{– University of Southampton (IRS, EPStats …)}

– University College London

–

…

• Germany (DINI / DFG)

Germany (DINI / DFG)

– Göttingen State and University Library

– Stuttgart University Library

DEUDEU

– Computer and Media Service Humboldt University Berlin

(18)

Germany

E.V . E.V .

y

W k h

E h

d

d Alt

ti

M t i

ION

• Workshop on Enhanced and Alternative Metrics

of Publication Impact, 20–21 February 2006,

b ld

l

O

Humboldt University Berlin

ÜR NETZ

W

ÜR NETZ

W

• Cluster of proposals to the DFG

– Network of certified open access repositories

2y

9 – Network of certified open access repositories

2y

9 – Usage statistics

demonstrator

9 Distributed open access reference citation service

TSCHE IN

I

TSCHE IN

I

– Distributed open access reference citation service

demonstrator

9

(19)

Germany II

E.V . E.V .

y

G

ll ti

i t f

i ht h

ION

• German collecting society for copyright charges

(VG Wort) has started a project on statistics

(

S)

(METIS)

• IFABC (International Federation of Audit

Bureaux of Circulation) standards for robot

)

detection and click spans (30 min)

TSCHE IN

I

TSCHE IN

I

(20)

Distributing VG Wort IDs

E.V . E.V . O RMA T ION O RMA T ION

Doc comes with or gets

VG Wort registers ID

if not known

O

Doc. comes with or gets

new VG Wort counter ID

e.g.

490271fca856ba65cc

VG Wort

PDF

_PDF

DNB

Repository ingest

Open access

Repository

ID is made available

through OAI PMH

DEUDEU

through OAI-PMH

(21)

Collecting data

E.V . E.V .

Repository

PDF

User

Rewrite Module

PDFaddress

Repository

PDF

User

Rewrite Module

mod_rewrite and

PHP

PDFaddress

PDF

http://vg00.met.vgwort.de/na/490271fca856ba65cc?l=PDFaddress

Log

Repository

Log

harvester

VG Wort

IFABC metrics

DEUDEU _COCOCO

(22)

Measuring impact:

the technical elements for usage data

E.V . E.V . OpenURL ContextObjects Log Repository Link

the technical elements for usage data

O RMA T ION O RMA T ION Link Resolver L CO CO CO Aggregated W ERKINF O W ERKINF O Data Metrics Log DB Log Repository Link Resolver CO CO CO Aggregated

Usage Data e.g.

ÜR NETZ W ÜR NETZ W Data Mining Filtering Services Log Log harvester (Service Provider) ITIA T IVE F ITIA T IVE F Filtering Aggregated logs g Repository CO CO CO Log DB Webserver -Log Aggregated Usage Data e.g. TSCHE IN I TSCHE IN I og g Rewrite OpenURL ContextObjects SUSHI Normalise COUNTER DEUDEU e e module

Normalise (optional) -> Robots, psydonymization or SUSHI

Based on: Bollen, Johan and Van de Sompel, Herbert, OAI4, Geneva

(23)

Conclusion

E.V

.

E.V

.

• Infrastructure for collecting and aggregating usage data

g

gg g

g

is conceptually available, has to be deployed and

implemented in practice on a large scale

• Investigating metrics for different needs and purposes

ÜR NETZ

W

ÜR NETZ

W

• Technical and social issues (the later posing the bigger

challenge)

– Standardization

– Modularity

– Co-Operation

p

– Openness

– Transparency

DEUDEU