How To Develop A Prototype For The Kbse

Full text


Critical Success Factors for Knowledge-Based Software

Engineering Applications

Barry Boehm and Prasanta Bose USC Center for Software Engineering

Department of Computer Science {boehm,bose}

April 1994

1.0 Introduction

Ten prototype knowledge-based software engineering (KBSE) applications were recently developed in a USC graduate course. These were expert-system applications falling into the Activity Coordination portion of the KBSA paradigm [Green et. al.,1983] rather than into the automatic program generation portion. The KBSE development guidelines for the prototypes included the primary critical success factor (CSF) heuristics cited in such references as [Waterman, 1986], [Jackson, 1990] and [Kelly,1991] for identifying potentially successful expert system applications. These included choosing applications with limited domains, with relevant expert knowledge available, and with relatively stable domain expertise. Following such guidelines, ten applications were selected and knowledge-based prototype systems were developed for them. Evaluation of the applications’ utility for practicing software engineers, showed differences that were not explainable by the above CSFs. In this paper we postulate an additional set of CSFs that aid in explaining the differences, and also provide insight into the key elements of the application that contribute to its utility.

2.0 Prototype KBSE Applications

The prototyped applications can be grouped into three software engineering task categories as follows:

i) Process assistance

ii) Software architecture and reuse assistance iii) Cost/schedule/risk assessment

Candidate applications in each category and associated knowledge sources were provided to the students, who could also define applications of their own choice. The prototype applications were developed by individual students or groups of two students with the


guidance of the authors. The students were also required to knowledge-engineer at least two domain experts in their selected application domain. The time to develop the application were restricted to one semester. Expert system shells on Sun workstations and personal computers (IBMs and Macs) were provided as candidate choices. The following subsections provide brief descriptions of the prototypes: the subtasks they handle, the rationale for developing automated or semi-automated approaches for the tasks, the approach taken, the scope of the prototype developed, and an evaluation of their utility to software engineering. Besides utility, the applications were evaluated with respect to clarity of the interface, and general robustness. The summaries below are distilled from the individual student reports. The evaluations of the applications were done by the authors and subsequently corroborated via demonstrations of the applications to local industry personnel.

2.1 Process Assistance

2.1.1 Process Model Selection

• Objective. Providing assistance in choosing the best software development process model given a specific process context and product context.

• Rationale. Typically when a development model is used, it is often chosen for abstract reasons, such as "that's the way we always do it," rather than trying to pick the development process that best fits the project.

• Approach. The developed application program assists a program manager to quickly and easily determine the best process model to use in developing a new software package.The approach taken is based on using a case base of contexts for different process models [Boehm, 1989], matching the current context against the cases in the library and choosing the best matched model.

• Scope. Limited to top-level process model choice. It does not consider cases where hybrid models are appropriate.

• Utility. High utility, based on the rationale that it provides realistic top-level diagnosis

of high-leverage decision situations.

2.1.2 Preliminary Design Review Preparation Advisor

• Objective. Providing assistance in structuring of software reviews and audits.

• Rationale. The review activity is complex, requiring coordination of multiple

reviewers. A tool for aiding the review process can be based on explicit review process models.

• Approach. The approach taken makes use of explicit process models of the review

process and review guidebook heuristics to provide advice on actions to take and documents to generate for preliminary design review.

• Scope. Limited to providing advice in the preliminary design stage.

• Utility. Low utilty, based on the rationale that it did not go beyond generally available procedures and broad common sense. It failed to highlight potential critical problems that can compromise the success of software reviews.


2.1.3 Process Reengineering Assistant

• Objective. Acquire process models from action history.

• Rationale. The problem of process model acquisition is important to process management

and process improvement.

• Approach. The approach taken is based on i) actions in the Unix environment are

declaratively modeled as a set of schemas, ii) the Unix-shell history file input is used to find the applicable schema instances, iii) the action schemas are ordered to form a process instance. Such execution ordering constraints are determined from the timestamp information present in the elements in the history file. The process instance is generalized by variabilizing and typing the resources used by the actions.

• Scope. Limited to activities recorded in Unix-shell history files.

• Utility. Low, based on the rationale that the method can reeingineer only very simple

processes involving a single agent and executing linear process models. It fails to handle complex non-linear interleaved processes and processes involving multiple agents.

2.2 Software Architecture and Component Reuse

2.2.1 Software Architecture Balance Diagnosis

• Objective. Provide assistance to designers of software systems in design and analysis of

complex software architectures vis. a vis. the overall balance in the design.

• Rationale. Such assistance is important in architecture level design since incorrect design

decisions and overlooked factors lead to faulty architectures that have imbalances. Knowledge of types of imbalances can be used to diagnose for such faults and hypothesize corrections.

• Approach. The approach to address this problem involves i) the software system

components and their connections are represented as a set of schemas and form elements of the working memory in a production system and ii) Imbalance detection rules are represented as a set of production rules.

• Scope. Detect imbalances local to a specified module in a system context. Does not do

any required design change prediction to remove imbalance.

• Utility. Medium, based on the rationale that the prototype identifies simple static balance

problems using a highly abstracted applications model, but is insensitive to dynamic balance solutions like load-reallocation and fault-tolerance mechanisms.

2.2.2 User Interface Design Assistant

• Objective. Assist a user-interface software designer to develop and integrate a consistent, quality user interface in the software development areas of Requirements Specification, Rapid Prototyping, Design Review and Integration, and Acceptance Testing.

• Rationale. Encoding of style guide rules brings Human Factors style rules to non-experts or less experienced personnel, allows the designer to build a higher quality interface by providing immediate checks and feedback, provides a stricter measure of design choices by


making the designer aware of design criteria and requirements, and makes it possible to automatically check for consistency across interfaces that were developed separately for integration into the same system.

• Approach. Rule based system comprising a combination of Procedural and Rule-based knowledge. Uses Guide's Internal language form (GIL) as intermediate representation of all GUI. Style constraints are coded as forward chaining rules which recognize style violations in the GIL representations and recommend candidate revisions to restore consistency [Bolcer, 94].

• Scope. Intended for use with distributed development of large-scale software with multiple graphical interfaces including Window, mouse, toolkit, and X based applications. • Utility. High. It identifies frequently-misused user interface capabilities: color, menu-structure, screen layout. It identifies incoherent styles when two independently designed interfaces get composed.

2.2.3 Software Library Code Reuse Assistant

• Objective. Provide assistance in a megaprogramming environment to optimally re-use software components from a component library [Gacek,1993].

• Rationale. Manually searching for components that fulfill a user’s needs maybe complex. In addition, users start with partial knowledge of the components that match their needs. • Approach. For selecting component for a single function: i) Find components that match the required function. ii) Rank the components based on required priority on non-functional constraints. For selecting components for conjunctive functions: i) Find components for individual functions independently. ii) The components for individual functions are tested for common data structure usage in order to allow composition of the components for the conjunctive functions. iii) The filtered candidate component sets are rank ordered based on degree of match against required non-functional constraints.

• Scope. Assist on the selection of both data structures and software components to be used

based on the users' requirements.

• Utility. Medium utility, based on the rational that the application covers only a small

number of Booch/ Grace type software components, though the advice provided by the system is sound.

2.2.4 Data Reduction Software Architecture Advisor

• Objective. Assist in the design and layout of off-line data analysis and reduction software programs.

• Rationale. Data analysis and reduction software follow domain specific software templates. The templates can be used to guide the design and analysis of the software.

• Scope. Generates initial layout of the target system, creates new modules and links

between modules and analyzes for any inconsistencies. Does not do code generation, detailed specification of the data stores, and, documentation generation.

• Approach. Interact with the user in a menu-directed fashion. Present module choices to the user. The user specified choices used to define a design. The design can then be


• Utility. High utility, based on the rationale that it provides good advice on classification

and configuration of components though it is based on a narrow application domain.

2.3 Cost/ Schedule/ Risk Assessment

2.3.1 Software Technical Risk Advisor

• Objective. Assess technical risks involved in a prospective software system. Provide top level risk resolution advice.

• Rationale. Many software system problems due to high-risk features becoming locked

into specifications through lack of awareness of technical risks.

• Scope. Assessment of software technology risks (vs. cost / schedule/ management risks).

• Technical Approach. The needs-vs-capabilities tables in DoD Software Technology Strategy Document [DoD, 1991] are coded as forward chaining rules. For a given project’s needs, the required capabilities are inferred. The required capabilities are used to determine the differences with existing current capabilities and compute the technical risks involved in choosing the current capability or evolving the capability. A weighted network is used determining risk prioritization for combinations of capabilities [Toth, 1994].

• Utility. High.It provides a level of risk identification and advice for very complex projects

which is generally available only from scarce domain experts.

2.3.2 Schedule Overrun Assessment

• Objective. Assist inexpert users in determining effects of external events on the slippage

of schedule.

• Rationale. Heuristic knowledge is used by experts to determine the factors causing

slippage and how to correct the slippage. Gathered knowledge is encoded in a system to assist non-experts in determining slippage.

• Technical Approach. The heuristic knowledge of the effect of a pattern of impactors on

the schedule are represented as production rules. For a given input schedule context, the rules are triggered. The effect of the actions of the rules lead to changes in the schedule parameters which consequently are classified as slippage.

• Scope. Makes use of a set of impactors and develops prototypical quantitative estimators

based on those impactors that cause slippage. Does not consider all possible interactions of the impactors on the schedule.

• Utility. Low. It provides quantative slippage estimates, but they are not causally linked to

several of the well-known sources of slippage.

2.3.3 Expert COCOMO Risk Assessment

• Objective. Develop a prototype knowledge-based software engineering assistant for cost

estimation and project risk assessment.

• Rationale. Expert knowledge can be employed during cost estimation activities by using

cost driver information for risk identification and assessment to detect patterns of cost and schedule risk.


• Approach. Knowledge acquired from various written and expert sources is represented as

forward chaining rules. The rules predict cost driver ratings based on a given project parameters. Cost driver ratings are used to index directly into tables of risk ratings. The risk weighting scheme uses assigned risk levels and cost multiplier data to compute overall risks for each category and the entire project [Madachy, 1994].

• Scope. The scope of the assistance is for project cost and schedule estimation and

associated project risks based on the COCOMO model and related expert knowledge. User inputs are evaluated for risk situations or cost model input anomalies and calculations are performed for the intermediate version of COCOMO. Limited advice is also provided.

• Utility. High utility, since it works on existing COCOMO cost drivers and provides

realistic project risk assessments.

3.0 Critical Success factors (CSFs)

As can be observed, the applications have differences in utility to practicing software engineers. In order to explain such variations we hypothesize a set of critical success factors (CSFs). We discuss below the success factors and the understanding they provide on the key elements in an application and how such elements contribute to its overall utility. Table 1 summarizes the results of the analysis of the utility of individual applications in terms of the individual CSFs:

• Core Decision-Driver Abstractions • Unambiguous Mapping to Abstractions • Critical-Mass Option Representation

Note that other potential explanations of the relative success of the applications were investigated, such as the developer’s length of industry experience, but these did not correlate well with the applications’ relative utility.

TABLE 1. Analysis of utility of the prototype applications based on the CSFs

Application by section number 1.1 2.2 2.4 3.1 3.3 2.1 2.3 1.2 1.3 3.2

Limited Domain + + + + + + + + +

Expert Domain Knowledge + + + + + + + + + Stable Domain Expertise + + + + + + + + + +

Core Decision-Driver Abstractions + + + + + + + + Unambiguous Mapping to Abstractions + + + + + + +

Critical-Mass Option Representation + + + + +

>2 years industry experience + + + + +


3.1 Core Decision Driver Abstractions

This CSF involves the identification and use of a core set of abstractions: i) which abstracts away the less critical aspects of the task domain ii) which maps well onto a software project's decision drivers and provides useful distinctions for project diagnosis and iii) which serves as a framework for the full set of rules and defines the solution space.

An example is the set of COCOMO cost drivers (e.g., product size, required reliability, development schedule compression, etc.) used as a basis for software project risk assessment in the Expert COCOMO application (#3.3). Two of the low-utility applications, the Preliminary Design Review Preparation advisor (#1.2) and the Schedule Overrun Assessment (#3.2), were not able to find good core decision-driver abstractions. The User Interface Design Assistant (# 2.2), was initially having difficulty in formulating rules, until the developer found a good set of decision driver abstractions: the GIL (Guide Interface Language) files used in Sun DevGuide interface development. These data structures provided an explicit linkage between the user interface designer’s decisions on color, menu layout, screen layout, etc., and the resulting look and feel of the user interface. The pre-GIL approach was trying to analyze sequences of graphic interface related system calls (e.g. draw-line, change-font, etc), which did not provide the needed framework for organizing rules and advice.

3.2 Unambiguous and Ease of Mapping from Facts to Abstractions

This CSF involves ensuring a consistent mapping from a given set of facts to the intermediate representation over which inference rules are defined. The other low-utility application, the Process Reengineering Assistant (#1.3) had a good set of core abstractions, but the complexity of its inputs (traces of software engineers' interactive sessions) created many unresolvable ambiguities in mapping the input facts to the abstractions. In most of the high utility applications, this CSF is met by choosing the problem description language same as the solution description language. For example, in the Software Technical Risk Advisor (# 3.1), the project needs are represented using the same terms as those used in the rules that model needs to capabilities mapping. This representational uniformity reduces the ambiguity as well as the complexity in mapping the input to an intermediate representation over which the inference rules are defined.

The Software Technical Risk Advisor, Expert Cocomo (#3.3), and the Process Model Selection Assistant (#1.1) were able to achieve this uniformity by obtaining the core decision drivers directly as user inputs. The Architecture Balance Assistant (#2.1) also used this approach, but was less successful for the reasons described in the following subsection.

3.3 Critical-Mass Option Representation

The more successful applications were able to address a critical-mass level of software engineer’s practical decision concerns. Some had a small number of rules, such the Process Model Selection Assistant (#1.1), but it was successful because the total number of unique, well-supported process models to choose from is not large. Others succeeded by addressing a niche area in a critical-mass way, such as the Data Reduction Software Architecture Advisor (#2.4) and the User Interface Design Assistant (#2.2).


However, some tools achieved tractability by simplifying their scope and making assumptions to the extent that they either failed on the critical aspects of the problem or failed to produce the more relevant solutions. For example, the Architecture Balance Diagnosis Assistant (#2.1), made simplifying assumptions about the sources of imbalances being only due to the static properties of the architecture and ignoring dynamic solutions to imbalances, such as load balancing for performance, mesh-size adjustment for accuracy, or redundancy exploitation for reliability. These solutions are popular enough to make a tool unattractive for use which does not consider them. Similarly, the Software Library Code Reuse Assistant (#2.3) addressed only a subset of Booch-type components for reuse, limiting its value to this relatively small part of an application.

4.0 Conclusion

We would conclude that the three additional CSF's identified (core decision-driver abstractions, unambiguous mapping from facts to abstractions, and critical-mass option representation) are indeed important for selecting knowledge based software engineering applications. Thus, the three CSF’s can be used as a filter for evaluating the potential return on investment of a candidate KBSA tool.

5.0 References

[Boehm, 1989]. B. W. Boehm, Software Risk Management, IEEE Computer Society Press, Los Alamitos, CA 1989.

[Bolcer,1994]. G. Bolcer, “User Interface Design Assistance For Scale Software Development”, submitted to KBSE 94.

[DoD, 1991]. “DoD Software Technology Strategy”, Office of the DDR&E, December 1991.

[Green et. al., 1983]. C. Green, D. Luckham, R. Balzer, T. Cheatham, and C. Rich, “Report on a Knowledge-Based Software Assistant”, RADC Technical Report, June 1983.

[Gacek, 1993]. C. Gacek, "KBSE Tool for Megaprogramming", IRUS Reuse-Based Software Engineering Symposium, September 13 1993.

[Jackson, 1990]. P. Jackson, Introduction to Expert Systems (2nd. ed.), Addison Wesley, 1990.

[Kelly, 1991]. R. V. Kelly, Jr., Practical Knowledge Engineering, Digital Press, 1991.

[Madachy, 1994]. R. Madachy, “Knowledge-Based Risk Assessment Using Cost Factors”, Proceedings, SEI Software Risk Conference, April 1994.

[Toth, 1993]. G. Toth, "Software Technology Risk Advisor", Proceedings, SEI Software Risk Conference, April 1994.



TABLE 1. Analysis of utility of the prototype applications based on the CSFs


Analysis of utility of the prototype applications based on the CSFs p.6