Assessing Model Predictive Performance

There are a number of strategies in assessing the predictive performance of the logistic regression model. These include consideration of the Akaike Information Criterion (AIC), the null and residual deviance, confusion matrices, and Receiver Operating Characteristic (ROC) curves. The AIC measures the goodness of fit of the models to the given dataset and estimates the quality for each of the models

compared to one another. It penalizes the model for added model coefficients so therefore models which minimise the AIC are preferred. Whilst residual deviance can give some indication of the goodness of fit of a binary logistic model, McCullagh and Nelder (1989) point out that this quantity cannot be used in this way, and does not have a chi-squared distribution. Confusion matrices are well known in statistical

126

learning and are described in Altman and Bland (1994). A confusion matrix cross- tabulates the actual vs. predicted values of the model after dichotomising the predicted probabilities into two categories “predicted reconviction” and “predicted non-reconviction” using a cut-off probability of 0.5 and assists in evaluating the accuracy of the model predictive performance. The counts of the true positives, negative positives, false positives and negative positives are displayed in a 2x2 table and the accuracy of the model is calculated with:

𝑇𝑟𝑢𝑒 𝑃𝑜𝑠𝑖𝑡𝑖𝑣𝑒𝑠+𝑇𝑟𝑢𝑒 𝑁𝑒𝑔𝑎𝑡𝑖𝑣𝑒𝑠

𝑇𝑟𝑢𝑒 𝑃𝑜𝑠𝑖𝑡𝑖𝑣𝑒𝑠+𝑇𝑟𝑢𝑒 𝑁𝑒𝑔𝑎𝑡𝑖𝑣𝑒𝑠 + 𝐹𝑎𝑙𝑠𝑒 𝑃𝑜𝑠𝑖𝑡𝑖𝑣𝑒𝑠+𝐹𝑎𝑙𝑠𝑒 𝑁𝑒𝑔𝑎𝑡𝑖𝑣𝑒𝑠.

Finally, another technique is the ROC (receiver operating characteristic) curve, which extends the idea of the confusion matrix. This is used for summarising and

visualising the models performance via a graphical output. It plots the sensitivity (the true positive rate showing the model’s ability to predict an event correctly) on the Y- axis and (1-specifity) (false positive rate) on the X-axis for various choices of cut-off probability. When the ROC is representing a perfect predictive model, the sensitivity is equal to 1 and specificity is equal to 0, with the plotted curve touching the top left hand corner of the graph. Therefore, the steeper ROC curve the better the predictive performance of the model.

The area under the curve (AUC), sometimes called the index of accuracy or

concordance index, gives the indication of the overall measure of fit of the model. It can be shown that the AUC also measures the probability that if a random pair of subjects (one true positive and one true negative) were chosen, the positive subject would have a higher predicted probability of the event in comparison the negative subject. Therefore, the AUC calculates the overall ability of the regression to classify between those offenders who reoffend and those who do not. An unpredictive model has an area of 0.5 and a perfect predictive model has an area of 1.

127

In Table 6.7 and Table 6.8 the AUC values have been calculated for all the logistic regression models and an example of the ROC curves are shown in Figure 6.1. The CCLS AUC values overall are slightly higher than the OI models, suggesting that the models predictive performance is doing better for CCLS dataset. The models are performing better for the 2 year follow up for both datasets and the B-Spline model AUC values are higher, suggesting that the B-Spline model is a better choice.

However, for the 10 year follow up, the AUC area values are lower (between 0.6-0.7) which are relatively poor predictions.

It is not really surprising that the ten year prediction models are poorer than the two year predictions as it is trying to predict so far into the future. If a developmental view of offending is taken, then life course variables and life -events such as gaining steady employment, marriage and parenthood will reinforce desistance from criminal behaviour and affect the offending outcome (Akers, 1999, Farrington, 1986,

Sampson and Laub, 2003). However, other factors such as unemployment,

alcoholism, drug taking, marrying another offender and association with law-violating peers may contribute to reoffending and intervene in this ten-year period and also affect the offending outcome (Akers, 1999, Tremblay et al., 2004, Lipsey and Derzon, 1999).

128

Table 6.7 AUC values for basic model

Offenders Index 2 year 10 year

Quartic4 B-Spline4 Quartic4 B-Spline4

AUC 0.677 0.688 0.6618 0.6628

CCLS 2 year 10 year

Quartic4 B-Spline4 Quartic4 B-Spline4

AUC 0.7104 0.7116 0.6628 0.6006

Table 6.8 AUC for models with gender

Offenders Index 2 year 10 year

Quartic4 B-Spline4 Quartic4 B-Spline4

AUC 0.678 0.6879 0.6139 0.6022

CCLS 2 year 10 year

Quartic4 B-Spline4 Quartic4 B-Spline4

AUC 0.7144 0.7156 0.6634 0.6643

Figure 6.1 CCLS 2yr quartic model with gender covariate – Example of ROC curve. Note that the x-axis is reversed in this plot.

6.3.1 Conclusion

Trajectory membership is a powerful predictor of reconviction for both the two year and ten year follow up periods. This method is a good alternative to other

129

reconviction methods which use other summaries of criminal history such as age of onset (Nagin and Farrington, 1992) or offending rate (Francis et al., 2007). The offending trajectory groups generated by the GBTM and BGBTM models have proven to be strong predictors of reconvictions. For both datasets, offenders belonging to the Class 4 (High Rate Chronic offenders) have reconviction

probabilities ranging from 46%-75% chance of reconviction in 2 years and from 69%- 91% chance of reconviction in 10 years. Those offenders in the OI dataset belonging to the ‘Low Rate Persistent’ group (Class 1 for the Quartic4 models and Class 2 for the B-Spline4 model) have reconviction probabilities from 6%-7% chance of

reconviction in 2 years and from 16%-17% chance of reconviction for 10 years. In the CCLS dataset those offenders belonging to the ‘Low Rate Persistent’ group have reconviction probabilities of 17% chance of reconviction in 2 years and 49% chance of reconviction in 10 years. Although there is variation in the reconviction probabilities between the two datasets, it must be noted that the datasets are different and

therefore the trajectory groups will differ and should not be directly compared. It would be sensible to expect some variation in the predicted reconviction probabilities because of this. The models perform better at predicting reconviction probabilities within 2 years than 10 years which is understandable as trying to predict so far into the future is difficult due to a number of factors which are discussed above. Using the AUC and ROC curve values to assess the predictive performance of the models shows that the 2 year B-Spline4 model is the best performing model, having the highest AUC values. This is encouraging for the use of B-Splines instead of polynomials in GBTM. Adding gender as a covariate in the model only marginally increases the models performance, suggesting that the effect of gender is already accounted for by the trajectory groups. This is possibly due to the fact there are very few females in either dataset and from the exploratory analysis in Chapter 4, it was revealed that males are the most likely to be recidivists.

130

The GBTM, BGBTM and logistic regression model results do not actually explain very much about the types of offenders that belong to both the datasets. It is unknown when or how many times each offender was reconvicted in the follow up periods. The dependent variable is only an indication of whether or not each offender had another conviction. The trajectory groups themselves are based solely on the number of offences committed over adolescence and early adulthood. No information about the type of offences is used or if offenders change the type of offences they commit. There could be some strong differences between the two countries if this information was used and may produce some very different offending groups if this was taken into account. The following chapters in this study begin to examine crime mix patterns and pathways using the types of offences through Latent Transition Analysis.

131

7.1 Introduction

The previous chapters focused on looking at the frequency of offending over the life course and how these frequency trajectories change with age. However, as already mentioned, this tends to ignore any changes in the patterns and types of offences being committed. Studying patterns of offending behaviour in detail is important for several reasons. For example, it allows the identification of which offence typologies appear to be precursors for other types of offences (Francis et al., 2004). Such information has both practical and theoretical implications. Knowing what types of offences criminals may commit prior to being involved in serious crime like murder, for example, can be of great importance to law enforcement agencies and policy makers as there may be scope for targeting such offenders before they move on to the more worrying pathways of criminal activity. Further, understanding possible links between various types of crime also has a theoretical force, which may enable to distinguish between different types of offenders. In short, gaining detailed knowledge of crime mix patterns – which is the main focus of this chapter – may well help with the understanding of offending behaviour and the causes behind it.

The goal of this chapter is primarily focused upon finding crime mix patterns and how they develop over the life course, within the two longitudinal datasets of criminal conviction histories. Using a latent Markov modelling approach, the criminal careers of offenders can be examined in more depth by exploring both the crime mix patterns of offenders and how these develop and change over time. By developing the

methodology used in Francis et al. (2010), where the idea of lifestyle specialisation and short-term crime typologies (crime mixes) over five-year age-periods was introduced, a latent Markov model can be applied to identify different crime mix

7 CHANGING CRIME-MIX PATTERNS OF OFFENDING

132

patterns and also estimate the transition probabilities from one age period to the next in the datasets.

It is worth a comment here about terminology. The terms of "latent transition analysis" and "latent Markov modelling" are both used in the literature for what is essentially the same model. The method has become more popular in the analysis of criminal career data as it shows the different phases of criminal careers and the transition between each phase. Thus Massoglia and McGloin et al have used the term Latent transition analysis (Massoglia, 2006, McGloin et al., 2009), whereas Bartolucci et al., (2007) used the term Latent Markov modelling. Which terminology is used seems to depend on the software used. Thus MPLUS and SAS users will tend to use the term Latent Transition analysis, whereas R, Latent Gold and other bespoke software will tend to use Latent Markov modelling. The term Latent Markov modelling will be used in this study except when referring to previous work.

This chapter proceeds with a discussion of crime mix patterns followed by the preparation and exploratory analysis of the datasets. This is then continued with a discussion of the methodological approaches and statistical analysis. Finally, the results of the Latent Markov Models are presented and discussed.