Best Practices for Running a Hyperfunctional Psychology Laboratory

(1)

Best Practices for Running a

Hyperfunctional Psychology

Laboratory

Greg J. Siegle, Ph.D.

University of Pittsburgh

School of Medicine

Presented work supported by MH082998 These slides available at

(2)

Why bother?

• You and others can trust your data

–

It’s easy to know when you step into a best-practices lab

–

Some researchers get a reputation as “careful”

• Increase replicability

• Decrease debacles

–

Example from my lab:

• The chilling chiller incident

(3)

Stuff we’ll discuss

• Study setup

• Data collection

• Storing data

• Analysis

(4)

(5)

Pre-emptive strike:

Clinical Operations Manual

INCLUDING template

documents

• Common elements

–

Study Responsibility Log – who

does what when

–

Study worksheet – stuff which

has to happen and when, e.g.,

calibrations, audits

–

Assessment Schedule

–

Assessment Grid

–

Procedural Checklists

–

Regulatory Binder Template

(6)

Regulatory Binders & Lab documents

• Basic clinical trial model

– 2 folders per patient – 1 for identifiable info, 1 for all study documents. + master list.

• Excellent list of lab documents

– http://www.uth.tmc.edu/ctrc/regulatory.html

– Binders/Folders for

• Protocol and amendments

• Data

– Subject Logs and Lists

– Patient Data – 1 per participant

– Contact Logs and monitoring

• Reporting

– Corrospondence with outside organizations (e.g., FDA)

– IRB Documents

– Case Report Blank Forms

– Adverse events

• People

– Investigator Information

– Team Information

• Lab information

– Lab certifications, etc.

– Equipment

– Investigational product (e.g., drug) info

• Meeting documents

– Study meetings

– Study reports

(7)

Study management database

Stuff to include

in addition to

data

• Subject information

–

Screening/Enrollment log

–

Visit Schedule Log

• Tracking/Reporting information

–

Adverse Event Log

–

Protocol Deviation Log

–

Data Cleaning log

• Accountability logs

–

Device calibrations and accountabilities

(8)

Quality management plan

• what

will you

check,

how

will you

check

it?

(9)

Quality assurance guidelines

(10)

Create folders

• Study folders: at least

–

data

• pupil

• heart

• behav

• ….

–

analysis

• matlab

• spss

–

documents

–

publications

–

regulatory

–

software

(11)

Folder contents (from Dr. Nicole Prause)

• Data

– Raw, important processed stages, data processing scripts such as .m file backup, compiled data, final data

– The data folder should contain enough information to quickly reconstruct

important phases of data processing without storing too many large files on the computer indefinitely.

– Every data folder should include is a "notes.txt" file, where you note

abnormalities for particular subjects and files to enable quick reconstruction of data sets. For example, if a person becomes ill and withdraws from the study, it will be much easier to find this noted in a single file than to start searching to understand why the last two test conditions are missing to make decisions about data inclusion/exclusion.

• Institutional Review Board Compliance

– Submissions, revisions, letters of approval, up-to-date informed consent

• Scripts

– Electronic questionnaires, up-to-date DMDX scripts, backup of stimuli if size reasonable

• Publication

– Poster presentations, papers being prepared, final drafts of accepted/published papers

(12)

Select protocols

carefully

• Stay as close as possible to

industry standards when

possible (deviating as

necessary…)

–

E.g., the Society for

Psychophysiological

Research has published

standards for EEG, ERP,

Startle, Heart rate, HRV,

EMG, disease transmission…

• http://www.sprweb.org/journa l/index.cfm

–

Internet questionnaires:

Skitka

• www.uvm.edu/~pdodds/files/ papers/others/2006/skitka200 6a.pdf

–

ASTM (standards body)

(13)

(14)

(15)

Procedural checklists & records

• Every detail is golden: Have checklists

and how-to guides

• Check the checklists

– “All records shall be prepared, dated, and signed (full signature, hand written) by one person and independently checked, dated, and signed by a 2nd_{person” (GMP (Good}

Manufacturing practices) 211.186)

• Electronic checklists?

– Possible

• “Electronic records may be considered trustworthy and reliable and be used in leiu of paper records provided that the electronic records have proper secuirty controls” (21 CFR Part 11 Subpart A Sec 11.1)

• “Ensure authenticity & integrity of electronic records such that the person responsible for the electronic record cannot readily repudiate the record as not genuine” (21 CFR Part 11 Subpart B Sec 11.10)

• Ensure the system can discern invalid or altered electronic records (21 CFR Part 11 Subpart B Sec 11.10 (a))

(16)

Trouble shooting guides

(17)

AFTER data collection

• Data cleaning

(18)

Video – more is better

• Essential for clinical interviews to at

least get audio. Video is better.

(19)

Task design

• Validation

–

Check timing / event logging

• w/ fMRI we test at the scanner 1x phantom + 1x pilot

before any protocol

–

Check single subjects

• Write analysis scripts for single subjects

BEFORE your first real subject

• Be a subject for your own protocols

• Test everything completely BEFORE your

first pilot subject.

• Test everything completely BEFORE your

first real subject.

(20)

Psychophys lab setup

• Neat reproducable lab setups

–

Diagram in your Ops Manual

to show how to do stuff

exactly the same every time

–

As many procedural diagrams as

might be useful

• Care about disease transmission

–

Bloodborne Pathogen control:

• Gloves – as much as possible

• Don’t abraid the skin more than you need to

• Disposable electrodes when possible

• Disinfect

– CIDEX if you have ventillation

– Control III + Suave shampoo if you don’t

• Wear a labcoat – that’s actually what they’re for

Dr. Nicole Prause’s lab setup

(21)

Checking stuff works before data

collection

• Protocols before your protocols

–

Check all communications between computers,

peripherals, and data collection devices

–

Make sure your stimuli show

• Have this in your checklists

• We have eprime routines to test

–

getting scanner trigger,

–

eye-tracker events

(22)

(23)

Data Security & Integrity

• Whitebox standards:

–

Keep original data in unalterable form

–

Have 2

nd

_{copy for any necessary changes (e.g., remove a few}

trials, concatenate runs…)

–

Ensure the system can discern invalid or altered electronic

records (21 CFR Part 11 Subpart B Sec 11.10 (a))

• Security

–

21 CFR part 11:

• Double password protection

• Standards

–

They exist for most things:

http://www.astm.org/

• IRB

(24)

Databases

• Huge science -

http://c2.com/cgi/wiki?DatabaseBestPractices

• E.g.,

–

Have primary keys

–

Don’t change schemas

–

Consistent long descriptive column names across tables

–

Try things first in a local database

–

Good rule of thumb: 20 columns per table – more is weird design

• Lab standards

–

Ids are in columns called “id”

–

All tables have id

• 21 CFR Part 11

–

Keep an audit history of date created and by who, and dates

changed/updated

(25)

Backups

• Ideally

–

Daily data backups

–

Weekly incremental computer backups

–

Monthly full backups

• Keep a set of backups in a secure place outside

your lab

(26)

(27)

Documentation

• Document everything

• Lab notebooks are essential

– Extreme: Open Lab Notebook

• http://en.wikipedia.org/wiki/Open_ Notebook_Science

• All work posted immediately to the public eye

• Good tool:

http://openwetware.org/wiki/Main_ Page

– Commercial approaches

• Big list at:

http://campusguides.lib.utah.edu/co ntent.php?pid=126157&sid=21316 70

– My approach: Powerpoints per

study

• Greg’s Journal template – on the PICAN server

– \\oacres3\rcn\pican\docs\gjsjourna l.pot

– Sharepoint blog?

– Database page for all changes with name, date, change

description

• Analyses should be reproducible

– I like 1 matlab or SPSS file with all commands that produce all analyses for a given study.

(28)

Reasons for using ELNs/

virtual workspaces

• 1. They are an efficient way of managing large projects, multiple

projects and multi-institution projects.

• 2. Provenance ensures that any accusation of fraud can easily be

addressed.

• 3. Addresses the problem of missing information due to turnover in

lab personnel (and students).

• 4. Can access research results from anywhere and therefore keep up

with the ongoing work in the lab while traveling.

• 5. These systems are already being used in industry, therefore are

studentsneed to be acquainted with them to be employable.

• 6. Meets requirements of granting agency mandates for data

managment plans.

• 7. Facilitates depositing data into data repositories for reuse and

repurposing.

(29)

(30)

Beyond Powerpoint

Lab Bench People layer

http://campusguides.lib.utah.edu/content.php?p id=126157&sid=2131670

(31)

Example commercial solution:

(Not endorsed just summarized)

• From labarchives.com

– Intuitive Electronic Lab

Notebook (ELN) organizes your laboratory data

– Preserve all your data securely, including all versions of all files

– Share information within your

laboratory

– Keep abreast of developments in

your lab even when traveling

– Collaborate with investigators by sharing selected data from your Electronic Laboratory Notebook

– Publish selected data to specific individuals or the public

– Protect your intellectual property

– Runs on all platforms, including Windows, Mac, Linux, iPad and Android devices Special classroom version of our Electronic Lab

(32)

Sample all-figures-in-paper script

%% associations of power change with change in other things within and between groups

ctrl=find((s.grp==1) & (s.usedids==1) & (s.nonadapt_power_on~=-999) & (s.nonadapt_power_on_day7~=-999)); cct=find((s.grp==2) & (s.usedids==1) & (s.nonadapt_power_on~=-999) & (s.nonadapt_power_on_day7~=-999)); tau=find((s.grp==3) & (s.usedids==1) & (s.nonadapt_power_on~=-999) & (s.nonadapt_power_on_day7~=-999)); cct_tau=[cct; tau];

fprintf('---\n');

fprintf('CCT r(power_on change, rumination change)\n');

st.r_powerOnChg_rsqchg_CCT=r(poweronchg(cct),s.rsqchg(cct),0,1,1.5,-999); figure(7); clf;

regplot(rescaleoutliers(poweronchg(cct)),rescaleoutliers(s.rsqchg(cct))); xlabel('Trial Frequency Power Post CCT - Pre CCT');

ylabel('Rumination (RSQ) Post CCT - Pre CCT');

figure(8); clf;

regplot(rescaleoutliers(poweronchg(tau)),rescaleoutliers(s.rsqchg(tau))); xlabel('Trial Frequency Power Post TAU - Pre TAU');

ylabel('Rumination (RSQ) Post TAU - Pre TAU');

figure(9); clf;

focindfromcct_change=-9.94-151.94.*poweronchg+109.13.*poweroffchg;

regplot(rescaleoutliers(focindfromcct_change(cct)),rescaleoutliers(s.rsqchg(cct))); xlabel('Unfocus Index Post CCT - Pre CCT');

(33)

Use best practices for

preprocessing data

• Again with the

Psychophysiology

guidelines

• http://www.sprweb.org/journal/index.cfm

• Visual inspection of artifacts

• When are artifacts ok to let in to data?

–

How much should we say we’re letting in?

• Contingency planning

–

What if you change preprocessing midway through?

• I think you should reprocess everything

–

What if you change preprocessing after-the-fact?

(34)

Quality control

• Diagnosis and Clinical

dispositions:

–

Case conferences

• Reliability on

ANYTHING subjective

• Double data entry

• See your “Research

Methods” textbook…

(35)

Check your data early and often

• Quality check psychophys data that day and

fMRI data within a week (while it’s on the

servers)

• Single subject analyses

(36)

(37)

Calibrations

• Regular – monthly calibrations of all

instruments

–

Currently done for pupilometer

–

Other stuff?

• MR center and BIRC have done calibrations,

e.g., stability checks regularly.

We don’t

request them. But we should for our own

documentation.

(38)

Security

• Double-locked file cabinets

• Password protection for computers, files, etc.

• Note: 21 CFR (Code of Federal Regulations) Part

11 - Food and Drug Administration (FDA)

guidelines on electronic records

–

has security standards for data

–

audits, system validations, audit trails, electronic

signatures, and documentation

–

http://www.accessdata.fda.gov/scripts/cdrh/cfdocs/cfC

FR/CFRSearch.cfm?CFRPart=11

(39)

Audits

• Every 6 months, all

data within that 6

months

• Quality management

help at:

– http://www.uthouston.edu/CT RC/trial_conduct/quality-management.htm

–

There are chart audit

tools

–

Regulatory file review

tools

• Every year – full audit

– should be easy

(40)

The human thing

• Laboratory mentality is important. Attend to it.

Anecdotal evidence suggests happy inspired

labs are often more functional.

• You will likely not be in touch with the

emotional health of the lab. Have someone

who is. Make their report to you on lab health

a regular thing.

(41)

Hire for your weaknesses

• Good labs often have people who are (not all

of these at once)

–

Detail oriented

–

Socially attuned

–

Tech savvy

(42)