Unreliability in the Data

Another concern with data collected from online samples is the degree to which individuals are attentive while responding to materials. Several researchers have

questioned participant attentiveness on MTurk (Rouse, 2015), but the prevailing conclusion has been that it is possible to obtain data from online samples that is as

reliable as data obtained from university undergraduate students (Buhrmester et al., 2011; Paolacci & Chandler, 2014). In order to identify the presence of careless responding in the data, several directed response items were administered throughout the study materials. Although this seemed to identify careless responders in one MTurk sample, there was evidence to suggest severe levels of unreliability in the data obtained from a second sample.

As documented in greater detail in Appendix E, several steps were taken to identify the source of this unreliability. These steps included setting stricter criteria for directed response items and duplicated cases, verifying eligibility with work experience, examining the validity of responses to open-ended questions, identifying multivariate outliers, and computing an index of within-person response consistency. The combined efforts of these approaches made it possible to extract a reduced sample which met

acceptable psychometric standards of reliability for all measures that were ultimately included in analyses. However, it is a major limitation of this study that such a large portion of the data needed to be removed, and that even by doing so some variables still had to be dropped from planned analyses entirely.

Knowing that varying levels of attentiveness was a concern in the second survey sample, analyses were presented at two levels of reliability. As discussed by Berinksy et al. (2014) this approach can help to balance the internal reliability and validity that comes with a reduced, attentive sample, with the external validity and generalizability that is associated with a larger sample. Importantly, the nature of the results did not vary substantially between the full and reduced sample across a variety of analyses (e.g., confirmatory factor analyses, descriptive statistics, zero-order and partial correlations, moderation analyses). These results suggested the stability of the nature of the findings despite the presence of some unreliable responding. Therefore, the results obtained from this second survey sample can serve as preliminary validation evidence for the LSA- Extended. However, it will be important for future investigations of leader job satisfaction to replicate these findings in more reliable and stable samples.

10.2.3 Common Method Variance

An important limitation of the current dissertation comes from the fact that across all samples, only self-reported data were collected. As such, there was the potential that CMV positively biased the association between reported variables (Spector, Rosen, Richardson, Williams, & Johnson, 2019). However, this limitation was known before data collection began. As such, a priori methodological design choices were made to ameliorate the effects of CMV.

First, the inclusion of variables in the validation of the LSA was carefully considered. Constructs that were most appropriately measured with self-report (e.g., cognitive ability, commitment, turnover intentions) were intentionally chosen for validation purposes. Those constructs that were most at risk of being inflated when assessed via self-reports (e.g., performance) were not included in the validation of the LSA (Lindell & Whitney, 2001). The decision to examine validation evidence of the LSA using only those constructs that were most appropriately measured by self-reports limited the nature of relations that could be explored, but in doing so minimized the likelihood that CMV would impact observed relations.

Second, only those constructs within the nomological network of job satisfaction for which there was substantial documented empirical findings were included in the examination of validation evidence. Wherever possible, the most popular and highly- researched measures were selected for each of these constructs (e.g., RSES). Limiting validation to these well-researched measures meant that for all tested hypotheses, the nature of associations in the present study could be compared to meta-analytic findings in the literature. As such, any deviation in the magnitude of relations between the present study and meta-analytic results could be used as an indication of the extent to which CMV or other sampling error variance impacted the results.

Last, all data in the present study were collected anonymously. As previously discussed, this likely meant there was little motivation for leaders to fake or respond dishonestly when completing the survey materials. As such, it was unlikely that social desirability bias would dramatically impact the results. Taken together, the

implementation of these study design choices made it unlikely that the data in the present study were vulnerable to CMV.

In addition to the procedural CMV controls advocated by Conway and Lance (2010), post-hoc statistical controls were also employed as a part of the current dissertation. In particular, Lindell and Whitney’s (2001) correlation-based marker variable technique was used, wherein all zero-order correlations between LSA job satisfaction and criterion variables of interest were adjusted to account for CMV effects. The degree of adjustment was determined by examining the correlation between all variables and a theoretically unrelated marker variable – Aesthetic Appreciation. The resultant partial correlations that were obtained after marker-variable adjustments were made were then tested for statistical significance. Interpretation of these partial

correlations allowed for a more conservative examination of validation evidence for the LSA and it was the implications of these findings that were discussed.

Despite these procedural and statistical controls for CMV, it is important to note a second limitation of the measures included in this dissertation. Each of the correlates examined as part of the validation of the LSA have demonstrated robust relations with job satisfaction in the literature. However, the constructs were self-referential and were not specific to leadership behaviour or leader-relevant outcomes. Given that one of the proposed strengths of the LSA is its capacity to assess the unique experiences of leaders, it will be important for future researchers to examine validation evidence of leader- specific facets with leader-relevant outcomes such as follower performance (Wang & Howell, 2012), leader identification (Kark, Shamir, & Chen, 2003), and follower-rated transformational leadership (Rafferty & Griffin, 2004).