PATIENT-CENTERED OUTCOMES RESEARCH INSTITUTE
FINAL RESEARCH REPORT
Testing New Ways to Measure How Patients
Rate Quality of Life
Bruce D. Rapkin, PhD1; Carolyn E. Schwartz, ScD2,3; Iliana Garcia, MPH1; Brieyona C. Reaves, MPH, CCRP1; Leah
Goldstein, LMSW4; Jie Zhang, MPH2; Tom Atkinson, PhD5; Yuelin Li, PhD5; Wes Michael, MBA6; Alex Sankin, MD7;
Mark Schoenberg, MD7; Marisa Cortese, PhD, FNP-BC8; Una Hopkins, DNP8; Seth Lerner, MD8; Bernard Bochner,
MD4
AFFILIATIONS:
1Department of Epidemiology and Population Health,
Division of Community Collaboration & Implementation Science, Albert Einstein College of Medicine, Bronx, New York
2DeltaQuest Foundation Inc, Concord, Massachusetts 3Departments of Medicine and Orthopaedic Surgery, Tufts
University Medical School, Boston, Massachusetts
4Department of Urology, Memorial Sloan Kettering Cancer
Center, New York, New York
5Department of Psychiatry and Behavioral
Sciences, Memorial Sloan Kettering Cancer Center, New York, New York
6Rare Patient Voice LLC, Towson,
Maryland
7Department of Urology, Montefiore
Medical Center, Bronx, New York
8White Plains Hospital Center for Cancer
Care, White Plains, New York
Institution Receiving the PCORI Award: Department of Epidemiology and Population Health, Albert
Einstein College of Medicine
Original Project Title: Development of Practical Outcome Measures to Account for Individual Differences
and Temporal Changes in Quality of Life Appraisal
PCORI Award Number/Project ID: ME-1306-00781 HSRProj ID: HSRP20143593
_______________________________
TABLE OF CONTENTS
ABSTRACT ... 5
BACKGROUND ... 7
Challenges to Assessing and Interpreting Patient-Reported Outcomes ... 8
Rationale for Assessing Criteria That Individuals Use to Appraise Quality of Life ... 10
Theoretical Foundation: The QOL Appraisal Model ... 12
Figure 1. Analytic framework for the QOL Appraisal Model ... 14
Findings From Preliminary Studies Based on the QOL Appraisal Model ... 15
Background on Bladder Cancer ... 18
STAKEHOLDER ENGAGEMENT ... 21
Types and Numbers of Stakeholders Involved ... 21
Methods Used to Identify and Recruit Stakeholder Partners ... 22
Methods, Modes, and Intensity of Engagement ... 22
Stakeholder Adoption of Research Evidence Into Practice ... 24
METHODS ... 26
Overview of Project Aims ... 26
Aim 1: Development of the QOLAP Version 2 ... 26
Aim 2: Development of the Brief Appraisal Inventory ... 31
Aim 3: Testing Concise Appraisal Measures in a New Bladder Cancer Sample ... 35
RESULTS ... 45
Study Sample Characteristics ... 45
Table 1. Demographic Characteristics of Study Sample (N = 110) ... 46
Descriptive Statistics for the QOLAPv2 and BAI ... 48
Table 2. QOLAPv2 Descriptive Statistics ... 49
Table 3. BAI Descriptive Statistics ... 54
Construct Validity: Item-level Analyses ... 55
Table 4. Correlations Between QOLAPv2 Items and Sample Characteristics ... 56
Table 5. Correlations Between BAI Items and Sample Characteristics ... 69
Table 6a. Principal Components Analysis of the QOLAPv2 and the BAI: QOLAPv2 Section 1. Definition of QOL ... 76
Table 6b. Principal Components Analysis of the QOLAPv2 and the BAI: QOLAPv2 Section 2. Combinatory Algorithm—Emphasis... 78
Table 6c. Principal Components Analysis of the QOLAPv2 and the BAI: QOLAPv2
Section 3. Personal Goals ... 79
Table 6d. Principal Components Analysis of the QOLAPv2 and the BAI: QOLAPv2 Section 4. Sampling of Experiences ... 84
Table 6e. Principal Components Analysis of the QOLAPv2 and the BAI: QOLAPv2 Section 5. Standards of Comparison ... 85
Table 6f. Principal Components Analysis of the QOLAPv2 and the BAI: BAI ... 86
Redundancy Analysis Results ... 89
Table 7. Canonical Redundancy Analysis of the BAI With QOLAPv2 Sections ... 90
Table 8. Prediction of QOLAPv2 Components Using Main Effects, 2-Way Interactions, and Higher-Order Cluster Patterns Based on BAI Components ... 93
Response Shift Hypotheses ... 96
Table 9. Correlations of the QOLAPv2 and the BAI Principal Components With the QLQ-C30 ... 98
Table 10. Correlations of the QOLAPv2 and the BAI Principal Components With the BLM-30 Scales ... 101
Table 11. Variance of BAI and QOLAPv2 Components by Antecedents (demographics and treatment site), Catalysts (life experiences), and Mechanisms (mood) at Time 1... 107
Table 12. Average Variance of 20 PRO Scores at Time 1 Explained by the BAI and QOLAPv2 in Combination With Antecedents (demographics and treatment site) ... 113
Table 13. Average Variance of 20 PRO Scores at Time 1 Explained by the BAI and QOLAPv2 in Combination With Catalysts (life experiences) and Mechanisms (moods) ... 115
Table 14. Average Variance of Change in 20 PRO Scores Explained by BAI and QOLAPv2 in Combination With Antecedents (demographics and treatment sites) ... 118
Table 15. Proportion of Variance Added by QOLAPv2 Components Over BAI Components in Prediction of 20 PRO Scales, in Combination With Antecedents ... 120
Table 15a. Average Increments in R2 Due to Inclusion of Effects Measured by QOLAPv2 After BAI in Prediction of PROs ... 120
Table 15b. Proportion of Total Increment in R2 Due to Inclusion of Each QOLAPv2 Section After BAI, Overall and by Specific PRO Domainsa ... 121
Table 15c. Relative Predictive Information Gained Over BAI per QOLAPv2 Item by Section, Overall, and by Specific PRO Domains ... 121
Table 16. Average Variance of Change in 20 PRO Scores Explained by the BAI and QOLAPv2 in Combination With Change in Catalysts and Mechanisms ... 123
Table 17. Proportion of Incremental Variance in PRO Scales Explained by
QOLAPv2 Components Over BAI Components ... 126
Table 17a. Average Increments in R2 Due to Inclusion of Effects Measured by the QOLAPv2 After Taking BAI Effects Into Account in Prediction of PROs, Overall and by Specific Domains ... 126
Table 17b. Proportion of Total Increment in R2 Due to Inclusion of Each Section of the QOLAPv2 After the BAI, Overall and by Specific PRO Domainsa ... 127
Table 17c. Relative Predictive Information Gained Over the BAI per QOLAPv2 Item Overall and by Specific PRO Domains ... 127
DISCUSSION ... 128
Measurement Development Aims ... 128
Enhancing Patient-Centeredness in Patient-Reported Outcomes Research ... 128
Working With Appraisal Measures ... 131
Potential Barriers to the Uptake of Study Results ... 133
Subpopulation Considerations ... 135 Study Limitations ... 136 Future Research ... 138 CONCLUSIONS ... 142 REFERENCES ... 144 RELATED PUBLICATIONS ... 149 ACKNOWLEDGMENTS ... 150
ABSTRACT
Background: Health-related quality of life (QOL) assessments are intended to capture the
patient’s perspective on health and well-being. Even if not articulated, differences in the ways people think about their health and well-being are inherent in all patient-reported outcomes measures. The impact of changes in health status and treatment on QOL can be attenuated, amplified, or obscured by these differences. In 2004, we developed the QOL Appraisal Model and related measures to describe differences in individuals’ criteria for evaluating their own QOL. Although challenging to administer, first-generation appraisal measures have been successful in accounting for individual differences and intra-individual response shifts in QOL. More user-friendly appraisal measures that can be readily administered alongside standard patient-reported outcomes are needed and would serve to advance patient-centered research and practice.
Objectives: To develop practical measures of patients’ cognitive appraisal of QOL, our study
aims were to (1) leverage existing data to identify important aspects of appraisal; (2) derive concise appraisal measures in conjunction with stakeholders who use QOL assessments for research, practice, or policy evaluation; and (3) test these measures in a sample of patients with bladder cancer. In this report, we summarize previously published reports describing the
creation of new appraisal measures and then present detailed findings regarding their performance and utility in a population affected by bladder cancer.
Methods: Through reanalysis of earlier data sets, stakeholder input, and 2 online surveys, we
developed 2 new measures, the 75-item QOL Appraisal Profile version 2 (QOLAPv2) and the 23-item Brief Appraisal Inventory (BAI). To validate and directly compare these measures, we conducted a study of patients treated for bladder cancer at 3 local centers. Along with appraisal, the study included well-established measures of global and disease-specific QOL, stable antecedents (demographics, social determinants of health), and time-varying catalysts (recent life experiences and present mood state), as dictated by the QOL Appraisal Model. We completed the survey on 2 occasions to examine changes in health and life events.
Results: We gathered appraisal and QOL data from 110 patients with bladder cancer by using
the QOLAPv2 and the BAI. Examination of appraisal item distributions and zero-order
correlations with QOL, antecedents, and catalysts indicated the need for further multivariate analyses. Our analyses examined how these 2 measures were related to each other and how they accounted for individual differences in QOL. We used principal components analyses to derive dimensions of appraisal from QOLAPv2 and BAI. Although we expected these 2 measures to converge, assessment of appraisal proved to be more complicated than anticipated. We found lower-than-expected overlap between these 2 instruments, although the ability to estimate QOLAPv2 components was strengthened when we included second-order interactions among BAI components. Both new appraisal instruments accounted for individual differences related to antecedents as well as response shifts due to catalysts across a range of QOL
Conclusions: Study findings support the validity and applicability of the QOLAPv2 and the BAI,
warranting further use of these instruments by stakeholders concerned with cognitive and motivational differences in patients’ responses to illness and treatment. These measures provide significant value to stakeholders by enabling them to systematically recognize and address patients’ personal criteria and standards for QOL in research and practice.
Limitations: Because of our in-depth focus on a single disease (ie, bladder cancer) and the
relatively small sample, this study must be considered a preliminary step in the validation of the QOLAPv2 and BAI. Comparing quantitative appraisal measures with cognitive think-aloud interviews is also warranted. Further research using these measures is needed to understand whether lower-than-expected redundancy between appraisal as measured by the BAI vs the QOLAPv2 indicates inconsistency in what the 2 instruments explain, or whether the BAI captures emergent aspects of appraisal that the QOLAPv2 cannot detect.
BACKGROUND
Health-related quality of life (QOL) encompasses an array of patient-reported outcomes (PROs), each reflecting patients’ subjective evaluation of an aspect of their health and well-being. Although QOL instruments have been developed using increasingly sophisticated methodologies, almost all are based on standard items. Such evaluative measures, including specific health-related PRO domains, more global QOL, and measures of satisfaction, provide no information about individuals’ frames of reference and standards of comparison for answering questions about their health and well-being; neither do they address the salience of items to particular individuals at a particular time. Accurate interpretation of self-reported QOL depends on understanding the actual criteria that patients use to appraise their own situation and how these criteria underlie and shape responses to standard measures.
Understanding the criteria that people use to respond to evaluative questions about QOL is fundamental to patient-centeredness. In this report, we discuss efforts to develop and validate instruments able to capture the particular criteria that individuals use when they report on their own QOL.
This study builds on 15 years of work based on a model of QOL appraisal that posits parameters for understanding the perspectives and assumptions that people bring to an assessment situation. Our model focuses on individuals’ frames of reference (eg, goals they want to achieve, responsibilities they want to let go); ways of recalling experiences (eg, the most recent, the most upsetting); standards of comparison (eg, other patients, one’s ideal health); and relative emphasis in reconciling discrepant experiences (eg, positive versus
negative, self-focused vs other-focused). We have conducted studies in populations affected by different chronic conditions and injuries using a combined qualitative and quantitative
instrument designed to assess each parameter of appraisal. In this earlier research, we have consistently found marked differences in the appraisal parameters that help to explain variability in people’s responses to emergent symptoms, surgical complications, and stressful life events.
However, the first-generation appraisal measure is complicated to administer and score. The original QOL Appraisal Profile (QOLAP) included 10 different open-ended questions that could take 15 minutes or more for a trained interviewer to collect. Verbatim responses had to be recorded and coded. The qualitative codes needed to describe these responses varied somewhat from population to population, with a minimum of 35 different codes required. Scoring involved data reduction based on the prevalence of different combinations of codes. This level of effort and complexity is simply not practical for most health outcomes studies. This PCORI project has appreciably advanced this area of research through the development of appraisal measures that are easier to administer and score in any research or clinical context where QOL and other related PROs are measured. Aims of this project were as follows: 1. Leverage existing data from earlier studies to identify empirically the aspects of appraisal
needed to explain differences in PROs within and across populations
2. Derive concise, new measures of appraisal by examining empirical findings with stakeholders who use QOL assessment for research, assessment of clinical practice, or policy evaluation
3. Test the reliability, validity, and ease of administration of concise appraisal measures in a new sample of patients with bladder cancer
This report summarizes published work to accomplish aims 1 and 2. Detailed analysis and discussion of aim 3, reported here for the first time, constitutes the bulk of this report.
Challenges to Assessing and Interpreting Patient-Reported Outcomes
Our focus on differences in patients’ appraisal stems from a critique of studies of QOL and PROs. Most constructs of interest in patient-centered outcomes research are anchored in individuals’ subjective evaluation, with no external point of reference. Such constructs include pain and symptoms, emotional well-being, functional status, role performance, adequacy of social support, and satisfaction with treatment as well as overall ratings of health.
Over the past several decades, many generic, domain-, and disease-specific QOL instruments have been developed. These assessment tools are almost exclusively
“nomothetic,” meaning they are based on the assumption that standard items are directly comparable across individuals and over time. Whether global or disease-specific, nomothetic items and scales convey little information about what different individuals’ ratings actually mean.
For example, when individuals rate their overall health as “good,” we cannot determine whether this means “good for a person with my diagnosis,” “good in light of my previously diminished expectations,” or “good compared with my ideal health.” We also cannot know whether their ratings refer to specific symptoms, recent news about their treatment, or an overall sense of well-being. Even ratings of specific areas of functioning such as difficulty climbing stairs are ambiguous without information about respondents’ particular expectations and standards of comparison regarding their abilities. Further, symptoms and functioning can vary, so individuals’ summary ratings regarding a given period of time may reflect their worst experiences, their most recent, or some subjective average.
Of course, instructing people to base their responses on circumstances or comparators might be possible, but these constraints may limit the ability to use the measure in different settings or populations. Many different subjective criteria might affect responses to most QOL items. Instructions to constrain even 3 or 4 of these criteria would be onerous and potentially off-putting to respondents.
Consider the kind of instructions that would be involved: “Rate how you felt about your
physical functioning this week. Please average over your best, worst, and usual days. Only take into account regular activities that you normally do or want to do. Make your ratings of each item by comparing yourself to what you think is typical for most people at your age.” Such
complex instructions run the risk of distorting the person’s experience of QOL if, for example, he or she tries to maintain a positive focus or makes comparisons with his or her own progress. Investigator-imposed constraints reduce the patient-centeredness of measures by requiring individuals to base their responses on situations and comparisons that may not be personally salient or meaningful.
The inherent and unavoidable ambiguities in QOL measurement have led to paradoxical and counterintuitive findings across different populations and health conditions. For instance, people with chronic illnesses often report QOL equal or superior to that for healthier people.1-7
Discrepancies commonly arise between clinical measures of health and patients’ own ratings. 8-10 Well-being measures do not consistently distinguish known patient groups, are often only
weakly related to objective criteria, and show little convergence across measurement
perspectives. All these inconsistent findings can be explained by underlying differences in the ways that people appraise their health and well-being.
Rationale for Assessing Criteria That Individuals Use to Appraise Quality of Life
QOL assessment is intended to capture patients’ evaluations of their health and
well-being. There is no inherently right way for individuals to think about QOL questions, and efforts to constrain how individuals think may distort their answers.
Even so, how individuals formulate responses does have considerable implications for outcomes research. Differences in the ways that research participants think about QOL can obscure the impact of changes in health status and the benefits or harms of treatment. The comparative effectiveness of treatments may depend on individuals’ subjective criteria for evaluating the outcomes they experience. For example, understanding how a surgical
intervention for cancer affects ratings of role functioning depends on the roles that individuals are considering when making their ratings.
Outcomes assessment becomes further complicated because individuals can change the criteria and standards they use to rate their health. Such changes, referred to as “response shifts,” often arise because of experiences with illness and treatment. For instance, criteria for rating difficulty in climbing stairs may change as individuals accommodate to advancing illness or anticipate recovery of stamina and mobility.
Research typically neglects differences in the criteria that individuals consider when rating their QOL, in effect relegating those differences to “measurement error.” Analyses that compare or predict QOL scale scores among individuals over time or across treatment
conditions contain no information about individual differences in the meaning or use of scales. Even QOL instruments developed using sophisticated psychometric techniques such as item response theory (IRT) and computerized adaptive testing (CAT) do not solve this problem.
Consider role performance and satisfaction as an example. Although items such as “I enjoy my job,” “I love my job,” and “I could not imagine doing any other work” might follow a consistent item-difficulty gradient, responses to these items provide no information about the criteria individuals use to appraise their role. In this regard, IRT depends largely on semantic relationships among the items. Individuals’ answers to this set of items will likely be consistent, regardless of whether their job satisfaction is related to travel perks, being part of a team, or having a laid-back office environment. To be sure, an estimate of a latent job satisfaction score from these items is likely to represent accurately and reliably the ways individuals would respond to similar job satisfaction items at that time. Yet, even with high internal consistency and desirable scaling properties, 2 people who give identical responses to these items do not necessarily experience their jobs in the same way. This is important because the ways in which changes in health status affect job satisfaction depend mainly on the specific criteria that
individuals use to evaluate their work roles. Physical disability may interfere with work involving travel more than it does a job at a laid-back office. These criteria are not measured in QOL assessment including item sets developed using IRT.
It follows that observed correlations of job satisfaction with observable catalysts like changes in health status will necessarily depend on the extent to which a sample includes people applying particular criteria. If most people in a sample rate their job satisfaction
according to opportunities to travel, then negative correlation with physical disability will likely be greater than if the majority are more concerned about having a laid-back job.
Research samples generally comprise unknown, heterogeneous mixtures of people applying these and many other criteria to appraise job satisfaction and other evaluative constructs. This heterogeneity leads to relatively weak and inconsistent associations between objective catalysts and evaluative measures such as job satisfaction or QOL because the same
between health change catalysts and QOL are statistically significant, aggregate effects substantially less than R2 = 1.0 mean that the experiences of many people in a sample are not
well described by the linear association. The challenge is finding the right variables to segment heterogeneous samples to determine how similar influences yield discrepant responses and how similar responses may arise for different reasons.
This argument is not a critique of QOL instruments themselves. One can reasonably assume that the evaluative ratings people provide on PRO items accurately reflect their feelings and experiences. However, these instruments are not intended to provide insight into the psychological basis for QOL ratings. Additional information is needed about the ways that people arrive at ratings of QOL and related constructs, to better understand how those ratings reflect individual differences and intra-individual changes in health.
Theoretical Foundation: The QOL Appraisal Model
To account for these interindividual differences and intra-individual changes in the self-evaluation of QOL, Rapkin and Schwartz11 developed the QOL Appraisal Model. This model
posits that any construct involving subjective evaluation involves 4 aspects of appraisal: (1) the goals, priorities, and concerns that comprise individuals’ frame of reference12; (2) the ways that
individuals sample experiences within their frame of reference that they deem pertinent to QOL13-17; (3) the standards of comparison they consider in evaluating these experiences,
including past history,18,19 perceptions of salient others, social norms,20,21 and personal ideals;
and (4) the ways that they formulate summary judgments or emphasize the salience of different experiences based on some combinatory algorithm.22 This 4-parameter appraisal
model has proven useful in understanding individual differences and longitudinal changes in health-related QOL.23
This QOL Appraisal Model was the basis for the development of the QOL Appraisal Profile (QOLAP), designed to describe the criteria that individuals consider when asked to evaluate their QOL. This first-generation instrument was broadly descriptive to explore the nature and influence of QOL appraisal. The QOLAP was designed to be administered after completion of standard QOL instruments, to elicit individuals’ reflections on the ways that they
answered preceding questions. This instrument includes 2 qualitative sections that assess individuals’ definition of QOL and the personal goals that matter most for their current QOL. We developed 2 sets of Likert-scale items to measure the kinds of experiences and standards of comparison that individuals considered in making their ratings. We also used a set of semantic differential questions to understand the kinds of experiences that individuals emphasized in making their ratings.
The QOL Appraisal Model, depicted in Figure 1, established a basic analytic framework for evaluating the role of appraisal in patient-reported outcomes research. The model can be adapted to look at individual differences as well as intra-individual changes in QOL. The
dependent variable in Figure 1 represents change in a QOL construct. Standard determinants of change in QOL include “antecedents” such as background demographic, cultural, and
socioeconomic status influences. “Catalysts” include health status changes, treatments, and other life events. Catalysts may influence “mechanisms” expected to influence change in QOL, such as social support, self-management, or coping skills. Antecedents, catalysts, and
mechanisms constitute a Standard Model for explaining QOL. As noted in Figure 1, partitioning variance in QOL that can be explained by the Standard Model is useful. Residual variance (bottom panel of the QOL dependent variable box) represents observed change in QOL that is discrepant from change predicted by the Standard Model (top panel of the QOL dependent variable box).
Figure 1. Analytic framework for the QOL Appraisal Model
As Figure 1 depicts, additional residual variation in QOL change may be explained by taking cognitive appraisal into account. Appraisal represents the criteria an individual applies to rating QOL. Antecedents, catalysts, and mechanisms can influence processes of QOL appraisal in ways that attenuate or amplify their effects on QOL. Health-related events can markedly change the ways people think about their QOL, and those changes will be reflected in the way they respond to QOL measures. Changes in appraisal may arise because of coping mechanisms, increased understanding of a diagnosis, avoidant thinking, habituation to stressful
circumstances, socialization to an illness-related identity, or self-blame owing to the nature of illness.
Response shift is an emergent construct in the QOL Appraisal Model, defined as a discrepancy from expected change in QOL that can be associated with a change in appraisal. The model distinguishes direct response shift, when residual QOL change is correlated with
change in appraisal as a main effect, from moderated response shift, when change in appraisal modifies or interacts with responses to catalysts or mechanisms. This model of appraisal and response shift has been demonstrated in a wide variety of studies (described in the next section); we applied it to guide all analyses in this project.
Briefly, analyses based on the QOL Appraisal Model proceed as follows: QOL change scores are the dependent variable in multiple linear regression (or the equivalent). Analyses employ a hierarchical order of entry. After controlling for the QOL pre-score, exogenous antecedent variables are entered followed by changes in health state, life events, and other catalysts. Variables describing mechanistic influences on QOL, such as social support or
treatment indicators, are distinct from appraisal,. The model is trimmed at each step to reduce the number of predictors in the equation. This order of entry accounts for variance in QOL change that the Standard Model can explain. One would examine effects related to appraisal by testing the predictive value of baseline measures of appraisal, change in appraisal, and
interactions between Standard Model predictors and appraisal measures. Response shift is indicated by a change in appraisal that explains residual variance in QOL, either directly or in interaction with catalysts, antecedents, or mechanisms. Total effects attributable to appraisal (without considering Standard Model effects) are also examined, to determine overlap
between variance explained by appraisal alone and variance explained by the Standard Model. Analysts should consider multicollinearity in interpreting effects, to understand how different aspects of appraisal influence QOL scores.
Findings From Preliminary Studies Based on the QOL Appraisal Model
In earlier studies using the original QOLAP, Rapkin, Schwartz, and colleagues11 found
that assessing appraisal clarifies individual differences in response to illness and treatment; this helps to explain discrepancies in individual change. For example, in a sample of 394 people living with HIV/AIDS, Li and Rapkin24 used classification and regression tree analysis to examine
how changes in the Medical Outcomes Study Short Form (SF-36) mental functioning over 6 months were related to changes in appraisal. Measures of “expected change” were calculated
negative discrepancies from estimates of QOL change (residuals) were associated with several distinct patterns of change in appraisal, such as changes in emotional focus (eg, ‘‘focusing on negative experiences and feelings’’) and subjective norms (‘‘comparing self with others”). This ability to explain residual change in QOL that remains after standard determinants are
controlled using changes in cognitive appraisal represents direct evidence of response shifts. Similarly, in a study of patients with multiple sclerosis, Li and Schwartz25 found that 20%
of patients demonstrated changes in appraisal that accounted for variance in change in QOL measured using the PROMIS (Patient-Reported Outcomes Measurement Information System) General Health measure, the Neuro-QOL Cognitive Function and Positive Affect & Well-Being short forms, and the Ryff Environmental Mastery measure. This study also found that response shifts depended on disease status. As hypothesized, response shifts were most prevalent in a ‘‘progressive disease’’ cohort, somewhat less in a ‘‘relapsing’’ cohort, and relatively rare in a ‘‘stable disease” cohort. Classification trees differed qualitatively across cohorts, suggesting recalibration, reprioritization, and reconceptualization of criteria for appraising QOL.
In the same sample of patients with multiple sclerosis, Schwartz and colleagues26 found
that individuals’ historical and current activities to build cognitive reserve were associated with differences in QOL appraisal. Although item response patterns differed slightly between
historical and current active measures of reserve, they generally reflected a tendency for high-reserve individuals to emphasize the positive and to focus on aspects of life that are more controllable, rather than idealized standards of comparison or goals that are unlikely to be attainable.
Schwartz, Finkelstein, and colleagues examined evidence for different aspects of response shift in evaluation of PROs for spinal surgery, measured using the SF-36 and the Life Satisfaction Questionnaire-11 (LISAT-11).27,28 They noted that response shifts caused patients to
use the same functional outcome report measure differently pretreatment and posttreatment. For example, patients who improved in terms of leg and back pain effectively “moved the goalpost” for rating QOL after surgery, and placed a greater premium on physical functioning. Conversely, patients for whom surgery had no effect reprioritized their ratings of pain to
include more social and emotional dimensions. In a separate analysis using the retrospective pretest, these investigators also found that implicit theories of change (ie, differences in patients’ conceptualization of the meaning of change) accounted for greater bias in evaluating surgical outcomes than did mere recalibration of scales.
Morganstern and colleagues29 interviewed 50 patients with bladder cancer before
surgery. Analyses included content coding of QOLAP personal goal statements, examining the relationship of goal attainment to content, and associating goal-based measures with the European Organization for Research and Treatment of Cancer (EORTC) Quality of Life 30-Item Questionnaire (QLQ-C30) scales. Associations among measures of goal content and goal attainment provided evidence for construct validity. Progress toward goal attainment
correlated positively with relationship- and role-based activity goals, but negatively with health concerns. Goals also differed according to age, gender, employment, and marital status, lending further support for construct validity. QLQ-C30 functioning and symptom scales correlated with goal content but not with progress toward goal attainment. This lack of correlation seems counterintuitive, because one might expect that poorer functioning would impede progress toward goals. However, findings suggested that patients’ goals may be determined or
constrained according to their levels of functioning, and they thus rated their progress relative to those goals. Alternately, progress toward goals may reflect a unique aspect of QOL that is untapped by more standard disease-specific scales like the QLQ-C30.
In a later study with an expanded sample of 215 patients with bladder cancer, Anderson and colleagues30 explored this issue further by examining the contribution that idiographic (or
self-defined) measures of QOL derived from the QOLAP made to global ratings of QOL.
Idiographic measures included perceived distance from attaining personally defined goals and difficulty associated with goal attainment. Standard measures of role functioning and future perspective explained 15.7% of the variability in preoperative global QOL. Including goal attainment and activity, difficulty explained an additional 12.0% of the variance. Smaller gains were seen on measures of global health, life satisfaction, mental health, and activity, suggesting
that idiographic measures capture aspects of QOL distinct from health and functional status defined by nomothetic scales.
These results and similar findings in other studies using the QOLAP led us to consider the need for a new generation of appraisal measurement. We were particularly concerned about replacing qualitative questions that were complex to administer and score and about reducing the overall length of the measure. We thought that reexamining these major studies as well as other work on appraisal and response shift would allow us to derive more
parsimonious approaches to the assessment of QOL appraisal. That is the basis for the present study.
Background on Bladder Cancer
Later in this report, we provide an in-depth evaluation and comparison of the properties of new appraisal measures in a prospective study of patients with bladder cancer. We chose to focus on bladder cancer because the disease and its treatment present many challenges to QOL. Each stage of illness has several treatment options, each of which can affect functioning and well-being in different ways. As such, QOL may vary markedly, depending on individuals’ experience of disease and treatment-related symptoms and their appraisal of those symptoms.
Bladder cancer is the fifth most commonly diagnosed cancer, with an estimated 74 000 new cases in 2015 in the United States alone. It is the second most common genitourinary malignancy; it is the fourth most common malignancy in men and the ninth in women.31
Depending on prognosis, treatment follows various pathways. Patients identified at an early stage, before cancer has invaded muscular tissue surrounding the bladder, may have the option of active surveillance. This involves repeated trans-urethral resection of bladder tumor (TURBT), in which a cystoscope is inserted through the urethra, the bladder is examined internally, and any local tumor growth is removed. Many patients also receive local treatment with Bacillus Calmette-Guérin (BCG), a tuberculosis vaccine that stimulates immune response in the bladder. TURBT, which may be repeated on an ongoing basis at 3-month intervals, has been associated with discomfort and pain as well as anxiety associated with disease progression.32
Despite the need for multiple invasive procedures, TURBT allows people to retain their bladder and preserve normal functioning for as long as possible prior to disease progression.
High-grade aggressive non-muscle-invasive bladder cancer and muscle-invasive bladder cancer are most often treated with radical cystectomy (ie, removal of the bladder) and
concomitant reconstructive surgery to create a urinary diversion. Some patients with less advanced disease also opt to have radical cystectomy to avoid the burden of ongoing
surveillance and reduce the possibility of disease progression. Despite survival benefits, radical cystectomy has major potential to negatively affect urinary, bowel, and sexual function, as well as body image. Treatment may also have broader ramifications for these patients’ QOL,
including social and role functioning, intimacy, self-esteem, and psychological distress.33
Difficulties that patients encounter are often determined by the type of urinary diversion they have.
Patients treated with radical surgery undergo 1 of 3 types of urinary diversions: ileal conduit (urostomy), neobladder, or continent cutaneous urinary diversion (continent reservoir). The decision of the type of urinary diversion depends on surgeons’ experience and patients’ performance status and comorbidities; thus, accommodating patients’ preferences may or may not be possible.
Of the 3 main options for urinary diversion, the simplest is ileal conduit. In this
procedure, surgeons connect the ureters to an isolated section of distal ileum and construct a skin stoma. A patient who has had an ileal conduit will need to use an appliance (stoma bag) for the rest of his or her life. Although this procedure may have marked impact on body image and daily activities, it has the lowest rate of early and late complications. As such, it is usually the best option for older and sicker patients.
The second type of urinary diversion is the orthotopic neobladder. In this approach, surgeons reconstruct a pouch from a section of small bowel and connect it to the urethra. Patients who have a neobladder must learn to void by straining. Approximately 20% of these
The third type of urinary diversion is the continent cutaneous urinary diversion. In this technique, surgeons construct a pouch from a piece of colon and connect a small stoma to the skin, usually at the umbilicus. Patients who have had a continent cutaneous urinary diversion must catheterize to empty this pouch every 3 to 6 hours for the rest of their lives.
Patients with an orthotopic neobladder or continent cutaneous urinary diversion have more postoperative difficulties and higher complication rates than those with an ileal conduit, but these procedures may help to preserve body image. All 3 different treatment options potentially impinge on everyday activities to differing extents.
Because continent cutaneous diversions do not involve a permanent external pouch, patients and physicians may assume that this approach will have better QOL outcomes than the ileal conduit. However, neobladders and continent cutaneous diversions have disadvantages as well. These reconstructions are technically more challenging and more time consuming to perform. Postoperatively, patients leave the hospital with indwelling catheters. Once the
catheters are removed, patients must undergo a period of education in the techniques required to care properly for the reservoir or neobladder. Patients are also at a higher risk for diarrhea, vitamin B12 malabsorption, and metabolic disorders. It may take up to 2 years for patients to regain adequate urinary control with a neobladder, so incontinence is an ongoing concern. Nevertheless, the QOL advantages of continent urinary diversions are believed to outweigh the potential disadvantages for patients undergoing radical cystectomy.34 This particular condition
poses many different issues for measuring QOL that our appraisal instruments are well suited to enhance.
STAKEHOLDER ENGAGEMENT
Types and Numbers of Stakeholders Involved
Throughout this project, we worked with a panel of 10 stakeholders representing developers, participants, and consumers of QOL research. This collaboration enabled us to ensure that the content, format, and procedures of the new appraisal measures were as concise and portable as possible without sacrificing key information. Stakeholders included individuals who have worked with study investigators in various capacities. Panel members are listed below, with further detail in Appendix A:
• Joanne Buzaglo, PhD, Vice President for Education and Research, Cancer Support Community, Philadelphia, Pennsylvania
• Joel Finkelstein, MD, Orthopaedic Surgeon, Sunnybrook Health Sciences Centre and University of Toronto, Ontario, Canada
• Mitch Golant, PhD, Senior Consultant, Strategic Initiatives, Washington, DC
• Nicole Hollingsworth, EdD, Assistant Vice President, Community and Population Health, • Montefiore Medical Center, Bronx, New York
• Wendy Kahalas, MA, Evaluation Specialist, New York State Department of Health AIDS Institute, Bronx, New York
• Maureen E. Lyon, PhD, Child and Adolescent Psychologist, Children’s National Medical Center, Washington, DC
• Sherry Schachter, PhD, RN, Director of Bereavement Services, Calvary Hospital, Bronx, New York
• Mirjam Sprangers, PhD, Professor, Medical Psychology, University of Amsterdam, The Netherlands
• Timothy Vollmer, MD, Director, Neurosciences Clinical Research, University of Colorado, Denver
• Elisa Weiss, PhD, Vice President, Patient Access and Outcomes, Leukemia & Lymphoma Society, Rye Brook, New York
Methods Used to Identify and Recruit Stakeholder Partners
We decided that the most productive step was to bring together people who could work well with one another and who were familiar with assessment of QOL appraisal. We wanted members who were likely to stay involved throughout the project so that they would become increasingly familiar with challenges in the assessment of appraisal. We also wanted
stakeholders who would be able to disseminate measures and champion appraisal assessment in their professional roles and affiliations. Several of the stakeholders (and investigators) are themselves cancer patients, survivors, or caregivers or are people living with other chronic, debilitating conditions.
Stakeholders had integral roles to play in this project. We wanted and needed to hear what stakeholders had to say about differences in how individuals appraise their QOL as well as the ways these appraisals change over time. We wanted to approach this from several vantage points: clinicians who must ask about QOL in their daily practice; health policymakers who need to understand the wide impact of programs; researchers who have struggled to make sense of complex QOL data; and, of course, patients and survivors who will have particular insights about ways that people interpret QOL questions in different contexts.
Methods, Modes, and Intensity of Engagement
Phases of Stakeholder Participation
To ensure meaningful involvement of stakeholders in this study, we introduced the panel to problems in QOL research and the rationale and methods for appraisal assessment. Rather than jump directly into interpretation of psychometric analyses, we first worked with stakeholders to develop and clarify their own perspectives on QOL appraisal. These early phases lay the foundation for deeper, more focused critical analysis of new appraisal measures. Throughout the project, we conducted 2-hour conference calls with stakeholders every quarter (see Appendix B for the Quarterly Agenda for Stakeholder Panel calls).
Stakeholder Impact on the Relevance of the Research Question
We worked with the core panel of 10 stakeholders during all stages of psychometric analyses and measurement development. The panel focused on ensuring that the content, format, and procedures were as concise and portable as possible without sacrificing key information. Through quarterly conference calls, this panel had input in writing and adapting items and specifying response formats and instructions. Working with the core stakeholder panel, we presented 4 early concepts for the new appraisal measure that were ultimately consolidated into a preliminary measure and submitted for review by a broader group of stakeholders.
Broadening the Stakeholder Base Via an Online Survey
To gain further input on measurement development, we asked panel members to identify additional stakeholders to review the draft appraisal measure and provide feedback via an online survey distributed using Survey Monkey. In addition to feedback about the measure, the survey also asked participants about their experiences with QOL research, their need to know about QOL appraisal in their clinical/advocacy work, the pros and cons of existing QOL measures, and whether and how they could accommodate more in-depth QOL appraisal measures.
Stakeholder input on participant recruitment.
To identify online surveyparticipants, core stakeholder panel members drew on their professional organizations (eg, Oncology Nursing Society), research societies (eg, International Society for QOL Research), public health agencies (eg, the Public Health Association of NYC), and patient participants in QOL studies (through the Bronx Community Cancer Coalition). Respondents received $100 for completing the survey. A total of 30 respondents completed the survey. We then coded responses and analyzed them for presentation to the core stakeholder panel.
Stakeholder input on study rigor and quality.
The 30 survey respondentsand multiple sclerosis. Nearly half of the respondents (48%) indicated that they use QOL measures to assess intervention outcomes. Of those respondents, 39% use QOL measures for needs assessments; 13% use them for comparing groups. Most respondents (75%) indicated that they use QOL measures for repeated assessments; 25% of the respondents use them only at baseline. A majority of the respondents (63%) endorsed the idea of “QOL appraisal” as very important to their work. Many noted that they would be interested in using a brief appraisal measure, once it was validated.
We also received useful feedback regarding the length of the instrument, clarity of instructions, literacy level, and anticipated challenges with specific patient populations (eg, adolescents and palliative care). Several respondents also indicated the importance of including religion or spirituality as part of the appraisal. The core stakeholder panel discussed this
concept extensively when reviewing survey responses.
Stakeholder Impact on Instrument Development
Based on feedback from the online survey, the new appraisal measure included 5 sections that addressed the 4 parameters of the Rapkin and Schwartz QOL Appraisal Model. Our original plan, as detailed in our application to PCORI, was to develop 1 new instrument with features similar to those of the original QOLAP. However, the concerns about length expressed by some online survey respondents led us to develop and test a second, briefer instrument as well. The Methods section gives details about the development of both new appraisal
instruments.
Stakeholder Adoption of Research Evidence Into Practice
In addition to providing input on the development of the new measures, Drs. Finkelstein, Hollingsworth, Vollmer, and Weiss adopted the use of 1 or more of the new
appraisal instruments into their work. Drs. Buzaglo, Golant, Lyon, and Schachter have discussed ways to incorporate appraisal into their own projects. Stakeholders’ firsthand experience with the measures provided useful perspectives on what measurement of appraisal brings to QOL
research, clinical practice, and policy. Stakeholders were particularly helpful in conceptualizing the format and clarifying instructions.
Several stakeholder panel members, particularly the physicians, asked that we not abandon the unique qualitative assessment of our earlier appraisal interview. They liked having the opportunity to understand patients’ concerns in their own words. Based on this feedback, we decided that abandoning the original QOLAP instrument was not necessary and that having several validated approaches to assess appraisal in different research and clinical contexts was most desirable. Rather than foreclose on a single tool, this project allowed us to compare a briefer option for the assessment of QOL appraisal with one that is more in-depth and descriptive, introduced in our Methods section.
METHODS
Overview of Project Aims
This project focused on developing measures of QOL appraisal, as a way of improving the validity and precision of QOL outcomes assessment. We sought to create practical measures for clinical and research purposes. Project aims were as follows:
1. Leverage existing data from earlier studies to identify empirically the aspects of appraisal needed to explain differences in PROs within and across populations
2. Derive concise, new measures of appraisal by examining empirical findings with stakeholders who use QOL assessment for research, assessment of clinical practice, or policy evaluation
3. Test the reliability, validity, and ease of administration of concise appraisal measures in a new sample of patients with bladder cancer
To address aim 3, the main focus of this report is a multisite study of patients with bladder cancer. Findings for aims 1 and 2 have been reported in 6 publications; these describe analyses that informed item content and format of the new appraisal measures and yield psychometric data obtained from onlinesurveys.35-40 Work related to aims 1 and 2 provided the
basis for the new measures of appraisal used in aim 3. We summarize these 2 aims below.
Aim 1: Development of the QOLAP Version 2
Creating Appraisal Items Based on Findings From Earlier Qualitative
Research
The goal of Aim 1 was to leverage existing data from earlier studies to identify empirically the aspects of appraisal needed to explain differences in PROs within and across populations. Building on stakeholder input and the earlier published work noted previously, the first aim of this project was to take a deeper look at our existing data to determine the best ways to summarize appraisal. We were guided by several major questions in this effort: Which aspects of appraisal are most important to assess? How can the number of questions be reduced or combined to reduce the time involved? Is there unnecessary redundancy among
sections of the QOLAP? What are the best item-formats to elicit appraisal? We carried out analyses to address these questions using our earlier data sets; these are documented in Appendix C and in a series of papers that highlight steps that we took to more fully explore the QOLAP data.30,36-42 Here, we draw on this work to summarize the steps taken to develop the
QOLAP version 2 (QOLAPv2), based on these results.
At the outset, we anticipated that we would need to replace the 2 qualitative sections of the QOLAP used to assess frame of reference: Definition of QOL and Personal Goals. In different studies using the QOLAP, we continually employed and adapted the same qualitative coding system to identify themes associated with QOL definition and personal goal attainment. As such, examining the prevalence and combination of codes pertaining to QOL definitions was relatively straightforward. In general, 11 to 14 thematic codes could account for patients’ QOL definitions across different studies. We used language drawn from patients’ responses to the QOLAP to write items based on these codes. Rather than necessitating a verbatim response, the new Definition of QOL section of the QOLAPv2 enabled patients to indicate how well their own definition of QOL agreed or disagreed with 18 different phrases corresponding to their own understanding of the meaning of QOL. Examples of phrases included the following: “Perfect health,” “Looking back on my legacy,” and “Having nothing to regret.”
Thematic content elicited by the original QOLAP question about personal goals was more complicated to distill into items than we had anticipated. Responses referred to
motivational themes and life domains as well as to specific relationships and/or situations. We wanted to capture aspects of goals that moderated the impact of catalysts on QOL. Thus, rather than basing items on the more common goal statements, we focused on the goals of some individuals, but not others, to ensure that that measure was sensitive to these differences.
To develop a representative set of items, we wrote “goal statements” that reflected respondents’ answers on the QOLAP. Goal statements were cross-classified according to 5 motivational themes (achievement, problem-solving, maintenance, acceptance, and
this grid, we selected (with stakeholder inputs) 25 goal statements to represent the types of personal goals that may or may not be relevant to individuals’ appraisal of QOL.
Items were rated on a 5-point scale indicating how well each statement reflected respondents’ own goals and concerns. We also included a “not applicable” option so that people could indicate goal statements that did not pertain to them. Examples of these items included the following: “There is a specific problem or challenge related to my finances, my living situation, or other practical matters that I want to resolve”; “I want to be able to
participate fully in important upcoming events”; “There is much more I want to accomplish at my job”; and “I want to stay in my current home for as long as possible.”
Sections pertaining to Sampling of Experiences (14 Likert-scale items) and Standards of Comparison (9 Likert-scale items) were retained from the original QOLAP. Sampling of
Experiences items ask individuals about the kinds of situations they recalled in responding to QOL questions and address several theoretical considerations that can enter into memory: valence/mood, primacy/recency, cueing, and the social demand characteristics of the interview. For example, we asked, “Did you find yourself thinking about the worst possible moments?,” “Did you try to remember everything relevant over the past 3 months?,” and “Did you try to give your first reaction to the questions?” Standards of Comparison items focused on points of reference considered in evaluating QOL, including “Most people your age,” “Your ideal or dream of perfect health,” and “A time in your life before you had this chronic condition.”
The final items probe the ways that individuals reconciled, combined, or emphasized discrepant experiences when they formulated responses to QOL questions. On the original QOLAP, we used a series of semantic differential items; however, this item format was not as informative as we had anticipated because a linear differential (eg, “Things that were
completed vs unfinished concerns”) did not work consistently; some people considered both poles equally, whereas others considered neither pole. We decided to rewrite these as Likert-scale items, consistent with the other sections of the new measure. This revised section consists of 9 items describing the QOL areas that individuals emphasized in rating QOL items, including the following: “There are so many ups and downs that it is hard to know what to
rate”; “You are more focused on things you have to do than on accomplishments”; and “You have been able to keep up your mood, despite problems.”
Examining Psychometric Properties of the QOLAPv2
In fall 2015, our team had the opportunity to administer the QOLAPv2 to more than 4000 patients recruited from the participant panels provided by Rare Patient Voice (RPV) LLC. RPV LLC helps pharmaceutical companies and other industries assemble large cohorts of patients with different chronic illnesses for drug trials. The leadership of RPV concluded that many of their patients would be interested in answering questions about QOL and appraisal. Analysis of this ample RPV data set allowed us to evaluate the psychometric characteristics of the new QOLAPv2 more extensively than would otherwise have been possible.
We obtained data on QOL appraisal and other measures from patients with multiple different diagnoses—far more varied than our previous appraisal studies. A total of 4173 members of RPV panels (out of approximately 31 000) responded to our survey, representing a broad range of chronic diseases (about 350 distinct diagnoses). This sample reported having multiple coexisting conditions, including, but not limited to, back pain (56.97%), cancer (51.93%), depression (50.78%), insomnia (44.86%), arthritis (42.15%), asthma (18.47%), and diabetes (11.33%). The average age of participants was 48, with a mean income of
approximately $50 000. One limitation of the RPV sample obtained was the age and sex characteristics. The vast majority of this sample was white (91%) and female (85%), consistent with RPV’s current membership base. Despite restrictions in demographic composition, we decided to examine psychometric properties of the QOLAPv2 as a prelude to subsequent validation in a more diverse sample, in terms of age and sex (aim 3).
Structure of the QOLAPv2
First-order principal components analysis (PCA) yielded 29 components across the 5 sections of the QOLAPv2 measure. We explored associations among different sections by conducting a second-order PCA.
These second-order components represent patterns of appraisal, pulling together first-order components from across the 5 different sections of the QOLAPv2. Patterns were
meaningful and reflected important aspects of QOL appraisal. Twelve second-order components pulled together different aspects of people’s frames of reference, recall of
experiences, standards of comparison, and sampling of experiences in ways that were coherent and interpretable.
These components suggested clear themes in the different ways that people evaluate their health and well-being. They included the following:
1. Wellness focus: Maintain a calm and healthy lifestyle; learn self-acceptance; keep up health activities and health care; focus on improvements; get used to the way things are; remain positive and balanced; do not think about the worst moments.
2. Health worries: Focus on what doctors say; high frequency of social comparison
3. Recent challenges: Recall recent relevant health episodes and challenges; learn to accept others; let go of self-expectations; make multiple types comparisons in rating QOL. 4. Spiritual focus: Emphasis on faith and generativity
5. Relationship focus: Interest in romance; improved relationships; self-acceptance
6. Maintain roles: Concern about personal accomplishments; Maintain community and work roles, address family responsibilities and problems, and deemphasize self-acceptance; concern about regrets
7. Independence: Maintain independence; resolve problems; remain in one’s home; have no regrets; resolve recent money problems and other negative circumstances; keep active; maintain participation in multiple roles.
8. Reduce responsibilities: Let go of responsibilities for a house or for other people; let go of self-expectations; spend time with family; do not think about the QOL issues raised by the survey.
9. Pursue dreams: Pursue dreams and goals; change living situation; do not base QOL ratings on comparisons to others one’s own age.
10. Anticipate decline: Prepare loved ones and living situation for declines in health; focus on ups and downs in health; focus on what the doctor said to expect.
11. Be worry-free: Compare oneself to others without health limits and not to others with the same condition; solve money living situation, and practical problems, rather than accept people and roles as they are; let go of self-expectations.
12. Lightness of being: Be spontaneous; do not complain about health; focus on health rather than illness; concern about how one is seen by others
Variance in Physical Functioning Explained Using the QOLAPv2
Based on these results, we decided to examine associations of second-order
components with different measures of QOL. By way of example, a series of regression analyses based on the QOL Appraisal Model (Figure 1) revealed that demographic antecedents and comorbidity catalysts explained 29% of the variance in physical functioning measured by the PROMIS Global-10. Conversely, personality factors explained 6% of the variance. In contrast, second-order appraisal components alone explained 35% of the variance. When we entered Standard Model demographics and comorbidities first into these models, appraisal accounted for an 11% increment in the R2 statistic (ie, 46% of total variance was explained).
These associations of QOLAPv2 second-order components with measures of QOL, as well as further analyses examining the association of appraisal and personality measures, led us to consider the possibility that appraisal might best be understood in terms of more integrated experiences or patterns of thinking represented by the second-order components rather than discrete parameters represented by first-order components. This set the stage for the next phase of this project.
Aim 2: Development of the Brief Appraisal Inventory
Derivation of New Items Describing Observed Patterns of Appraisal
evident in the second-order PCA, we were able to create a more concise, 23-item instrument called the Brief Appraisal Inventory (BAI). This new inventory uses a measurement strategy that differs in important ways from the QOLAPv2.
Specifically, the QOLAPv2 maintained the molecular (ie, individual or small components) distinctions among the 4 main parameters of appraisal in the QOL Appraisal Model. We wrote the BAI items to reflect molar (ie, larger or overarching components) and emergent patterns of appraisal observed in the QOLAPv2 second-order components analysis, as well as findings of our earlier analyses on the “essence of appraisal.”37 The BAI asks respondents to consider the
extent to which each pattern of appraisal reflected what was on their mind when they responded to the QOL measures they had just completed. Initial drafts of BAI items were reviewed with stakeholders and subjected to cognitive interviews with 5 respondents before initial psychometric testing.
Rationale for Assessing Appraisal in Terms of Patterns of Ways of Thinking
The BAI items reflect combinations of appraisal parameters that captured the most variance across the QOLAPv2 domains. In writing the BAI items, we drew on much of the language of the source items so that statements were as close as possible to the original combinations in meaning. When combinations of variables loading on the QOLAPv2 second-order components were potentially too complex or ambiguous, we broke these out into 2 separate items. The resulting BAI items were written to reflect things that an individual might be thinking about during an interview. The 23 items may be understood as sampling broadly from the universe of thoughts that individuals may be having as they complete a QOL questionnaire.
This led us to put a caption at the top of the measure asking, “What’s been on your mind?” This approach used natural language that does not require individuals to drill down introspectively to somewhat more abstract questions about their cognitive experience removed from context. Nonetheless, we recognized that this sample of items could not capture all the detailed self-reflection represented on the lengthier parent measure (QOLAPv2). The question
was whether the BAI could account for large enough proportions of variance attributable to appraisal and response shift in meaningful ways, sufficient to warrant its inclusion in future research. We sought to examine the tradeoffs between these 2 approaches in aim 3 analysis.
Psychometric Properties of the BAI
We administered the newly developed BAI to a second cohort recruited from the RPV online panels. As with the original RPV sample, members of RPV’s online panel of
approximately 31 000 patients with chronic disease received an email invitation to participate in this study. We used this data set to conduct a psychometric analysis of the 23-item BAI. A total of 592 volunteers completed the BAI along with other measures (mean age 43.8, SD = 18.5; 79% women).
In terms of item characteristics, a missing item analysis revealed that 95.1% of
respondents completed 19 or more items of the 23-item BAI. The remaining 4.9% of respondents who completed 18 or fewer items were omitted from the analysis. The final sample included 563 respondents who responded to 19 or more items. Participants dropped from the analysis because of missing BAI data tended to be men.35
The BAI stem asks respondents to consider the list of 23 items in order to indicate “How much was each question on your mind as you completed the survey today?” Responses are based on a 5-point Likert scale (1 = never, 2 = rarely, 3 = sometimes, 4 = often, 5 = always). Because we retained only those cases with 4 or fewer missing items, any sporadic missing items were assumed to be “not applicable” and were recoded as “never on my mind.”
Frequency analysis revealed that respondents used the full range of the response options on every item. Of these 23 items, 52% of responses had a negative skew toward the “always” response item; 48% had a positive skew toward the “never” response item. The mean response for each individual item ranged from 2.0 to 3.7, with an overall mean of 2.9 for all items.
Structure of the BAI and Relationship of Appraisal Components to QOL
The 23 BAI items demonstrated many highly significant inter-item correlations (p < 0.001), prompting us to try to reduce this measure further. PCA with Varimax rotation yielded 5 components that explained 59.7% of the total variance. Item communalities ranged from 46% to 76%. We characterized the 5 components as follows: (1) health worries; (2) interpersonal and independence concerns; (3) accomplishing goals and problem-solving; (4) calm, peaceful, and active; and (5) spiritual growth and altruism.
We examined the correlations of the 5 BAI components with the demographics of the second RPV sample, comorbidities, health-related QOL, and NEO Personality Inventory (NEO-PI)43 with scores for openness to experience, conscientiousness, extraversion, agreeableness,
and neuroticism (see Rapkin et al35 for details on measures used in this study). It should be
noted that although the NEO-PI scale names reflect traditional understanding of personality traits, the items themselves offer a straightforward way to examine self-concept.
• Health worries had the largest number and magnitude of correlations with the other constructs, particularly with worse physical and emotional functioning. People who endorsed health worries tended to be female, younger, and sicker, with low past social support, low conscientiousness (eg, likely to describe oneself as lazy and not thorough), and higher neuroticism (eg, likely to describe oneself as nervous and not relaxed).
• Interpersonal and independence concerns were correlated with physical functioning. People high on this component also tended to be sicker and have lower levels of education. • Accomplishing goals and problem-solving was not associated with QOL subscales. People
endorsing this appraisal pattern tended to be male, younger, less likely to have cancer, and more likely to endorse depression. They also reported lower current social support and registered as less conscientious on the NEO-PI.
• The calm, peaceful, and active component was correlated with better physical functioning. People endorsing this appraisal pattern had fewer comorbidities, were less likely to endorse back pain, had higher levels of perseverance and current social support, and were more
agreeable (eg, more likely to describe oneself as trusting and not finding fault with others), as well as conscientious and less neurotic.
• Spiritual growth and altruism was unrelated to the QOL subscales. People endorsing this appraisal pattern tended to be female, with lower levels of education and higher
perseverance scores.
Aim 3: Testing Concise Appraisal Measures in a New Bladder Cancer Sample
Aim 3 of this study was to test the reliability, validity, and ease of administration of the newly derived, concise appraisal measures in a new sample of patients with bladder cancer. The present study focuses on comparing and validating the QOLAPv2 and the BAI in a prospective study of diverse patients treated for bladder cancer. We conducted a series of analyses focused on construct validity to examine whether these measures demonstrate anticipated
demographic differences in appraisal. In accord with hypotheses based on the QOL Appraisal Model, we also tested whether these measures function as mediators and moderators of the impact of changes in clinical status and other stressful events on QOL. We explored the association of appraisal assessment with patients’ mood states to examine hypotheses regarding the role of mood in the appraisal process. In all these analyses, we examined the extent to which the BAI could serve as a stand-in for the longer QOLAPv2. In the following section, we describe the results of this prospective study as well as details of our hypotheses.
Study Design
This prospective study collected data from a diverse sample of bladder cancer survivors recruited from 3 sites: Montefiore Medical Center in the Bronx, New York (Montefiore);
Memorial Sloan Kettering Cancer Center on the Upper East Side of Manhattan (MSKCC); and White Plains Hospital in suburban Westchester County immediately north of the Bronx. Data were collected at 2 time points separated by 3-month intervals. The study protocol is in Appendix D. Self-report instruments and the cognitive interview script are in Appendix E.
December 31, 2012, and June 30, 2017. Participants had to speak English or Spanish, be 18 years of age or older, and be able to provide informed consent. Exclusion criteria included not having follow-up care at the recruiting site or having metastatic disease at diagnosis (or both).
Study Outcomes
Research staff administered the following self-report measures and clinical information in addition to the new appraisal.
Quality of life.
Two measures used were from the EORTC questionnaires. The genericEORTC QLQ-C30 core measure evaluated 6 major domains (physical, role, emotion, social,
cognition, and global QOL); 3 symptom scales (fatigue, pain, emesis); and 6 single items (financial impact, dyspnea, sleep disturbance, appetite, diarrhea, and constipation).44 The
bladder cancer–specific
