Validation and Replication

Overview

Definitions of validation and replication

Difficulties and limitations

Why?

False positive results still occur…. even after stringent QC, data

pre-processing, complex analyses and alpha adjustments

The best ways of ensuring an observation is in fact real and

meaningful is to:

• validate and replicate the findings

• perform longitudinal and functional studies to determine the true causal/biological effects

Validation vs. Replication

Validation

Verify that the methylation data generated are accurate and the

results are reliable

Ideally, by repeating the experiment in the same samples but

using different laboratory techniques

Several factors could result in erroneous data. For instance:

• experimental design issues (e.g. cases and controls on separate plates) • handling errors (e.g. sample mix-ups)

Validation enables you to ensure the findings are due to true

biological variation and not some unknown experimental

artefact

Replication vs. Validation

Replication

Reproduce the findings in a independent dataset, i.e. different

samples

Replication enables:

• verification of the findings in a different dataset

• the findings to be generalised to the wider population • a more precise estimate of the findings to be measured • further exploration

The ideal scenario

Perform both

Validation proves the results are reliable but not necessarily

generalisable to the wider population

Replication, if successful, proves the results are generalisable

But, if unsuccessful, you will not know why

• technical error in the first and/or second stage • lack of power in the second stage

• subtle sample/phenotypic differences

In reality

Its not always possible to do both

• Sites of interest may not be feasible on certain platforms • Limited access to tissue samples

• Limited access to similar phenotypic cohorts

• Application of different study designs e.g. parent-offspring pairs, monozygotic twins, longitudinal studies may not be possible

Any attempt at validation and/or replication is better than

Summary so far

Validation:

Verify that the methylation data generated are accurate and the

results are reliable

• same samples, different method

Replication:

Reproduce the findings in an independent dataset

Validation and replication are not the same thing, but both are

valuable tools

Examples from our group

We have utilised a number of different processes:

Repeat the experiment in the same samples using a different methodology Repeat the experiment in the same samples using a different source of tissue

but the same technique

Include extra samples to increase robustness Assess different measures

(e.g. expression, methylation, SNP genotypes)

Independent replication i.e. different samples but same experimental method and study design

LHON is a common mitochondrial disorder characterised by loss of central vision

Hypothesis: Oxidative stress arising from mitochondrial dysfunction alters DNA methylation of the nuclear genome with consequences for the regulation of gene expression

We measured DNA methylation of the nuclear genome using 27k array to identify differences between those with LHON phenotype and

unaffected carriers

• Samples from four pedigrees from the North East of England.

Example 1. Leber’s Hereditary Optic Neuropathy (LHON)

Identify methylation differences associated with Leber’s hereditary optic neuropathy

Identify methylation differences associated with Leber’s hereditary optic neuropathy

UK family pedigrees with Leber’s hereditary optic neuropathy

Hannah Elliott, ongoing Discovery 27k chip Identify differentially methylated CpG sites (n=28) Blood samples 2 CpG sites selected to take forward (p<0.05) Bisulphite modification & Pyrosequencing of 2 candidates (n=28) Methylation levels strongly

correlated (rho >0.6) between techniques and

trends in association for both genes (p<0.1) Validation Blood samples Bisulphite modification & Pyrosequencing of 2 candidates (n=49) With an additional 19 samples mainly from the

same families, one candidate remained associated (p=0.006) the

other did not (p>0.1)

Validation/Replication Blood samples Replication Independent cohort French family pedigrees Bisulphite modification & Pyrosequencing

microarray expression analysis to identify genes with differential expression in preterm-born children defined as slow or rapid growers.

• Identify potential candidates for methylation analysis

Example 2. Postnatal growth and DNA methylation are

associated with differential gene expression of TACSTD2

and childhood fat mass

Postnatal growth and DNA methylation are associated with

Postnatal growth and DNA methylation are associated with

differential gene expression of TACSTD2 and childhood fat mass

CHILDREN BORN PRETERM: Newcastle Preterm birth cohort

Alix Groom et al, Diabetes 2012

Blood samples 11yrs

expression microarray slow vs rapid postnatal

growth (n=20) RNA

Validation of top hit using Real time PCR

Analysis of relationship between methylation, expression and phenotype at age 11y

Bisulphite modification DNA

Pyrosequencing analysis of candidate gene (n=94)

Saliva samples 11yrs

DNA

Bisulphite modification

Pyrosequencing analysis of candidate gene (n=68)

Postnatal growth and DNA methylation are associated with

differential gene expression of TACSTD2 and childhood fat mass

CHILDREN BORN TERM: ALSPAC

Alix Groom et al, Diabetes 2012

Analysis of relationship between methylation and phenotype at age 9 and 15 years

Blood samples 7yrs DNA

Bisulphite modification

Pyrosequencing analysis of candidate gene (n=178) Cord blood samples

Bisulphite modification DNA

Pyrosequencing analysis of candidate gene (n=173)

177 individuals from the population-based epidemiological ESTHER study: current smokers, former smokers, and those who had never smoked

Illumina HumanMethylation 27K BeadChip

Smoking and methylation

Smoking and Methylation

177 individuals from ESTHER study Discovery 27k Chip Identify differentially methylated CpG sites Blood samples 1 CpG site selected to take forward Bisulphite modification & Sequenom EpiTYPER analysis of discovery samples Spearman correlation between methods: (rho =0.82) Smokers still hypomethylated at CpG site (P_smoking = 1.07x10-28₎ Validation Blood samples Bisulphite modification & Sequenom EpiTYPER

analysis of 328 non- overlapping subjects Pronounced association

with smoking remained

Replication Blood samples Looked at methylation in surrounding regions using Sequenom EpiTYPER 79 samples from the discovery study

Further discovery

Only CpG sites immediately next to

the main hit were associated with smoking (41bp away)

…They then went on to test the same methylation site in a different cohort (Better replication?)

• Sequenom EpiTYPER analysis

• This time looking at whether F2RL3 methylation was related to a clinical outcome

1206 individuals from the KAROLA prospective cohort study

• Experienced acute coronary syndrome, myocardial infarction or coronary intervention

• Active follow up over 8 years

Methylation at F2RL3 associated with mortality in patients in this cohort

! The methylation data (CpG_4) reported in the main body of the paper IS NOT the same CpG site described in the original paper. This CpG is “CpG_2” – see

supplementary data for results The strongest signal from the first round wasn’t the strongest

association when linked to clinical outcome in a second cohort

Conclusions

Validation and replication are different

Ideally, attempt to do both

Plan for further functional work or analysis to identify true

causal/biological effects

If you can….

References

Smoking, F2RL3 methylation, and prognosis in stable coronary heart disease

Breitling LP et al., Am J Hum Genet. 2011 Apr 8;88(4):450-7. Epub 2011 Mar 31:

Tobacco-smoking-related differential DNA methylation: 27K discovery and replication

Groom A et al., Diabetes 2012 Feb;61(2):391-400. Epub 2011 Dec 21:

Postnatal growth and DNA methylation are associated with differential gene expression of the TACSTD2 gene and childhood fat mass

Hirschhorn JN and Daly MJ. Nat Rev Genet. 2005 Feb;6(2):95-108:

Genome-wide association studies for common diseases and complex traits

Rakyan VK et al., Nat Rev Genet. 2011 Jul 12;12(8):529-41. doi: 10.1038/nrg3000: