Assessing a Policy Management Framework

The four major categories of cost that we consider relevant are as follows: (1) setup, (2) interpretation of policies, (3) implementation of constraints, and (4) extracting compliance guarantees.

Setup

Most database programmers are familiar with writing triggers, constraints, and assertions. Thus, there is practically zero setup time in the “no-framework” approach and a program- mer can begin interpreting a policy document over the schema without delay. However, in

the context of our proposal, artifact definitions have to be agreed upon and various frag- ments of CLTL have to be made ready in visual notation for different departmental policy makers. Depending on the complexity of the organizational workflows and the database schema, this process can either be time consuming or trivial.

There are some quantifiable metrics during setup that can be identified as relevant indicators of overall cost. For example, the amount of overlap among artifacts and the average number of workflows applicable on a single artifact definition represent information that could be of interest to future researchers. These indicators are invariably related to the business and how its data storage requirements impact the structural complexity of the database schema. Similarly, the type, structure, and formulation of CLTL constraints preferred by system administrators can also provide insight into how users who may not be familiar with temporal logic respond to the framework. We have to keep in mind that most of the complexity associated with CLTL is hidden to the policy makers through the use of visual notation. However, the DBA or the equivalent supervisory team that decides on a particular fragment of CLTL may itself have to learn about temporal logic and eventually make decisions on notation, types of constraints, and the assumptions under which the framework will be presented and explained to individual policy makers.

Since most quantifiable determinants of cost and complexity are situation-specific (i.e., dependent on policy makers, system administrators, and the complexity of the business), it may be difficult to assess their correlation to the overall setup cost and make generalized statements about their impact. Most notably, individual biases towards certain types of constraints, processes, and workflows in one industry can lead us to make incorrect broad judgements that may not be applicable in other situations. Nonetheless, we hypothesize that if a comprehensive real-life examination of the usability of temporal logics in busi- ness situations is conducted, many of these soft indicators of complexity and cost can be identified and discussed in greater detail.

Interpretation

In Chapter 3 we argued that there are many advantages to everyday business users being empowered to express their policies on top of database level artifacts. Intuition suggests

that business users (not necessarily programmers) have intimate knowledge about policies, and if they can directly implement or modify constraints on a database, then significant cost savings can be realized.

To verify our intuition, it is essential to separate the notion of cost during the policy interpretation phase into two components: (a) reduction in incorrect interpretations, and (b) speed or ease of interpretation. In the traditional approach a programmer, if uncertain about the interpretation, must interact with a business user familiar with the given policy for guidance. Thus, the process is not only slower but arguably incurs a higher risk of errors through the involvement of multiple parties. Similarly, we note that high level business policy documents that contain descriptions and specifications of constraints rather than their formal interpretations are best understood by non-programmers. So a business user seems to be best qualified to read policy documents and interpret them in an error-free manner.

Unfortunately, to verify the above hypothesis is far from easy. Putting the “no- framework” approach against an artifact and LTL-constraint-system approach requires that we conduct an analysis of the interpretation capabilities of a programmer against those of a business user. Furthermore, we assume that this business user has now been properly trained in developing constraint systems and that this experiment is conducted in a controlled situation. This assessment is extremely difficult to replicate across several situations. Ideally, we want to be able to make claims about the correctness of an average programmer’s interpretation of a policy document on a database against those made by a business user over a set of pre-defined artifacts and to identify how long the process takes in each case.

Implementation

An examination similar to that presented in the interpretation phase extends into imple- mentation. Once again, we want to pit a programmer against a policy maker. However, as alluded to earlier, at and beyond this phase, the infrastructure of a comprehensive policy modeling framework will likely outperform any manual approach. Practically all met- rics ranging from run-time efficiency of code to time taken in development and testing of

constraints will favour an automated code generation system geared towards a particular database engine.

Nonetheless, it is important to measure this cost so that it can be offset against the initial setup time. Note that, from a cost savings perspective, the programming hours that are saved during implementation should be much higher than the administrative hours during setup. We have to draw out this distinction since different users cost a business different amounts per hour of effort. Thus, an accurate assessment of time saved requires that we measure the time taken by programmers to create triggers for enforcing business rules derived in the previous phase against a time of near-zero in the automated approach.

Maintenance

Maintenance of a set of policies and business rules can be described as the removal of certain constraints and/or the introduction of new ones. The process of maintenance is ongoing and not different from repeating the interpretation and implementation phases. Note, however, that the one-time setup cost incurred under the artifact/LTL-constraint framework now gets further amortized over the lifetime of a business and the database. Consequently, we need to test the hypothesis, first presented in Chapter 3, that the marginal cost to make changes to a policy/constraint system in the “no-framework” method is much higher (and progressively increases with the complexity of the policy set) when compared to the same marginal cost in the artifact/constraint-LTL approach.

Compliance Guarantees and Supporting External Audits

The final aspect in which the usefulness of our proposal needs to be evaluated is in its ability to generate compliance guarantees and to provide some level of assurance to internal and external auditors.

When examining the related problems of governance, risk, and compliance (GRC) in Chapter 2, we observed that it is often difficult to predict what auditors will look for in an given database. Thus, it is imperative to self audit and ensure that there are no inconsistencies within corporate processes. Chapter 5 subsequently revealed that there are

no standardized notions of correctness, inconsistencies, and conflicts when artifact level policies interact. Thus, internal corporate compliance officers in conjunction with DBAs need to rigourously test and subsequently prove that the constraints on a database can never lead to undesirable consequences.

Since we cannot predict what an auditor will be looking for, the question of which properties warrant formal verification is best left for experts in corporate GRC. However, we can focus on the question of how to complete the verification process. Formal verification of hand written code is challenging but still accomplishable. Our proposed system, on the other hand, has significant advantages of automating this process and reducing it to satisfiability and model checking (k-satisfiability). However, many questions about artifact constraint systems and verification remain unanswered. For example, how large do we expect these “typical workflows” to be, and how many variables, conditions, and types of functions will be involved in the state definitions. Earlier we had stipulated that these workflows rarely exceed 100 or so states, and analysis of a 1000 states represented an extreme case. Such statements would either be supported or refuted in a longitudinal study.

There are several other issues that were not addressed when evaluating the efficiency of generating proofs in our framework. Foremost is the issue of an upper bound for k when the notion of k-satisfiability was introduced in the context of bounded model checking. We alluded to the fact that workflows typically have a maximum “diameter” that essentially represents the longest possible non-looping sequence of state changes in which an artifact can partake. Thus, it would be reasonable to consider that diameter as an upper bound for k. However, without actually having a solid understanding of real-world user-generated constraint systems, it is difficult to directly assess the feasibility of this upper bound. If the time required to model check properties over workflows exceeds user expectations, then certainly a wide variety of issue open up and many possible optimizations and simplifica- tions may have to be examined. Along the same lines are questions pertaining to the interpretation of unsatisfiability results, that is, how best to explain these results to users (e.g., interpreting the least unsatisfiable cores) and the possible fixes for conflicts.

An important metric associated with compliance is that of policy coverage. Coverage is a measure that divides the number of policies that a system can implement and reason over

by the total number of policies to which a business is subject. Thus, it is important that we measure the proportion of constraints that can be covered in our framework relative to those that would require a more complex logic for reasoning (e.g., calendar and periodicity constraints). This would provide significant insight into the applicability of a system built around linear temporal logic and possible future extensions.