Evaluation

My research evaluation considers two concerns, introduced at the beginning of this research: Can I provide automated recognition of changes in patient behavior, where a patient’s behavior is defined by the stream of events that they initiate with the AT?

Can I provide automated diagnosis of a patient’s behavioral change, as characterized by the most prominent behavioral differences around a moment of change?

Through software construction, testing, experimentation, and case study, I affirm both propositions. Next, I consider two more issues: (1) model quality and (2) validation by real- world events.

Accuracy and Precision

Accuracy and precision are standard metrics for determining predictive model quality. As Table 3 shows, the read model is very good and while the compose model is useful not as good. This may be an anticipated because the read model depends on received emails and free time of the patient; these elements are mostly routinized for the user population. On the other hand, email composition depends on the patient’s skills and interest in communicating. Such interest depends on the content of the received email as well as external, non-modeled events, such as the correct usage of medicine or the arrival of visitors. In a prior study, I showed that patient interest and email composition increased with the addition of new email buddies, while decreasing slowly thereafter [30]. Therefore, given the limited information in the data available and the real- world behavioral variations of the users, the availability of at least one very good model appears to provide adequate information to identify some significant behavioral changes.

54

Real-World Events

Changes in model quality reveal changes in user behavior. My case study illustrates this with the inflection points that Table 5 and Table 6 summarize. Note that there is a causality chain from the user’s manipulation of the AT to the diagnosis of goal transitions:

 A user exercises the AT interface, such as reading and composing email.

 Data mined models are generalized from and accordance with the distribution of the events.

 Model differences are calculated, revealing changes in the models, which reflect changes in the event distribution, and thus changes in the user behavior.

 Model differences, at goal transitions, are characterized according to the events observed. Thus, assuming the algorithms and software are correct, there is a direct causal chain from changes in usage of the AT and the diagnosed changes presented to clinicians.

I discussed the results with the other TAL researchers. The goal transitions do seem to reflect persistent changes in behavior. One example may be revealing. Week 34 of the data corresponds to 8/20/2006 - 8/26/2006, while week 82 of the data corresponds to 7/29/2007 - 8/4/2007 (week 31 of 2007). I hypothesize that something interesting happens to the patient in the August summer holiday, such as a family member visit. Patient anonymity prevented me from directly correlating such real-world events, but it has been intimated that such events have occurred.

Threats to Validity

I believe the design science objective of extending existing theory and creating new technology has been met [50]. Through testing, performance analysis, and case studies I have demonstrated the effectiveness of the techniques for discovery and diagnosis of goal transitions.

55

A review of the related research supports the novelty of using model quality and model differencing of stream-mined models. The main validity concerns are the accuracy of goal transition identification and generalizablity of the approach, which of course, is a ubiquitous research concern.

Transition identification can be tuned using parameters of model quality. The analyst can select the metrics (e.g., accuracy, precision) that specify the quality metric, q, as well as how multiple models qualities are compared against the threshold to indicate a potential transition. The following illustrates the rule that if two models, with an intervening model, have a negative change in quality then consider the intervening model a transitional model:

If (mi.q'<ε and mi+2.q'<ε) then consider mi+1 a potential transitional model

The analyst must tune the transition identification parameters. Too many potential transitions will consume too much effort from the clinicians, who use the results to evaluate the clinical plan. Too few potential transitions will result is missed opportunities to adapt the AT or train the user. Therefore, the analyst must work with the clinicians to tune software to an appropriate threshold. In the TAL research, clinicians find the notification level reasonable and useful in their work of monitoring dozens of patients.

Generalizablity of the approach is another scientific concern. Model quality and model differencing is appropriate for patient behavioral monitoring, as demonstrated here. That alone is a significant contribution. However, can model quality and model differencing be applied effectively to other domains, such as business activity monitoring or network analysis? This open

56

question has provided me a new research opportunity in the area of financial business activity monitoring.

Consider the problem of monitoring personal credit card transactions, which is similar to the problem I am undertaking. Although simpler than rehabilitation goals, users do have goals for using credit cards, such as short-term payment, long-term payment, and insured (rental car) transaction. Financial models can be stream mined from transactional data. Behavioral changes can be inferred from model quality changes and characterize via model differences. Consequently, as a user transitions from short-term financing to long-term financing, the techniques described herein can automatically flag and characterize the change—at least my initial efforts suggest that this may be appropriate. It’s fair to say that analyses of stream-mined models, such as quality metrics and differencing, provides research opportunities for real-time data analytics.