Auerbach Publications
© 1999 CRC Press LLC
INFORMATION MANAGEMENT: STRATEGY, SYSTEMS, AND TECHNOLOGIES
C
RITICAL
F
ACTORS
IN
D
EVELOPING
A
D
ATA
W
AREHOUSE
Duane E. Sharp
I N S I D E
Are Companies Realizing A Return On Their Investment?, Internal Access To The Corporate Data Warehouse, Building the Data Warehouse, Focusing on the Real Problem, Selecting the Right Data Warehouse Champion,
Using Detailed Historical Data, Applying Technology, Trust The Data
INTRODUCTION
Data warehousing has become one of the most significant technologies of the past decade, and has permeated virtually every business sector, from retailing to finance, in one form or another.
The International Data Corporation (IDC) estimates that revenue from the total worldwide data warehouse software market, including data ac-cess, warehouse management/storage, and data transformation/ware-house generation, will grow at a compound annual growth rate (CAGR) of 30.8%, during the period from 1995 to the year 2000. Worldwide mar-ket revenue was $1.4 billion in 1995; based on this forecast, it will grow to $5.4 billion by the year 2000. This growth pattern is a certain indica-tion that data warehousing is well beyond the stage of early adopindica-tion and has been accepted by pragmatic businesses as a proven technology for enhancing their business operations.
As an example of the proliferation of this major corporate application of information technology, it is worth noting that NCR Corporation, a world leader in data warehousing technology, had over 500 data warehousing in-stallations worldwide in 1997, in a
broad range of business sectors.
ARE COMPANIES REALIZING A RETURN ON THEIR INVESTMENT?
A recent IDC ROI study, published as “The Foundations of Wisdom: A Study of the Financial Impact of Data
P A Y O F F I D E A
Although the past decade’s experiences have proven that data warehousing can provide a com-pany with high ROI, developing a warehouse is one of the most complex projects a company can undertake. This article outlines a five-step pro-gram for increasing the likelihood of success when developing a data warehouse.
Warehousing” by Stephen Graham, interviewed 62 sites that have suc-cessfully implemented a corporate data warehouse. The study covers a wide range of industries, including financial services, health care, tele-communications, retail, government, and manufacturing. The average initial investment by the surveyed sites was $2.2 million. The major find-ing of the study is that organizations recouped their initial investment within an average of 2.3 years. The average return on the initial invest-ment over 3 years was more than 400%, dramatic confirmation that data warehousing can be a good investment.
INTERNAL ACCESS TO THE CORPORATE DATA WAREHOUSE
A data warehouse takes time-oriented data from multiple applications and organizes it according to subjects meaningful to the corporation or business. Corporations, concerned with informing their decision makers, are pursuing this strategy for two major reasons:
1. Reduced complexity. The data in the decision-support database or ware-house is made available in a form that is relatively easy to understand. 2. Improved performance. The warehouse can be tuned to provide
bet-ter performance and fasbet-ter response to complex queries and analysis.
BUILDING THE DATA WAREHOUSE
From a qualitative perspective, according to the IDC survey, the key ben-efits of a corporate data warehouse are:
• More streamlined systems administration; and • More productivity for internal analysts.
Building a data warehouse is one of the most complex processes a cor-poration can undertake. It will change the corporate decision-making pro-cess without nepro-cessarily reengineering the corporation. Traditionally, corporate decisions have been based on the analysis of data, without de-tailed information to support the data. Corporations analyzed data from re-ports and made decisions based on limited information. Data warehousing changes this process dramatically, by quickly transforming all available de-tailed data (irrespective of volume) into meaningful business information. The end results are timelier and better informed business decisions.
Experience based on successful data warehouse implementations points to five critical factors, which are essential for a successful imple-mentation. The following analysis of these factors decribes why they are important to any data warehousing project.
Focus on a Real Problem
It is a fundamental axiom thata successful data warehouse implementa-tion needs to be based on solving a real business problem, and the
cor-poration will have to solve this problem. A data warehouse which does not address a critical business problem is destined for failure.
The business problem selected must have senior management back-ing which correlates with the desire to solve the problem. Most success-ful data warehouses are cross-functional, because the ROI increases with both the breadth of data they hold and the impact they have on the busi-ness. Business problems that have been solved by a data warehouse so-lution include:
• Credit card risk management; • Sales and inventory management; • Supply chain management; • Exposure management; and • Target marketing.
Solving these business problems requires large volumes of data from many business functions and, in some cases, even from outside sources. It also involves structuring the data based on input from end users who will use the system, as to what data is important and how it should be presented.
History has shown that if a data warehouse is built without end-user input, end users will not use it and the development exercise will be a spectacular, expensive failure. Information technology professionals alone cannot build a data warehouse: user organizations must be in-volved from the beginning.
There are several approaches to implementing a data warehousing system. One solution which is often applied to solving a business prob-lem is the so-called packaged data warehouse. A packaged solution is usually a single-vendor solution, with a pre-defined front-end applica-tion, a standard database management system, and an industry-generic database design. It often fails because it does not solve the critical busi-ness problems of a corporation; however, it is implemented to prove the concept. Since it is designed to meet a variety of requirements, it usually fails to address the specific needs which are always a part of any organi-zation’s data warehousing business.
The data warehousing solution which is most likely to be successful is one that provides a solution to critical business problems, specific to the organization for which it is designed, with significant end-user involve-ment and senior manageinvolve-ment support.
Select the Right Data Warehouse Champion
The second critical success factor is acquiring a strong champion for the data warehouse implementation. The complexities of the implementa-tion are enormous, ranging from maneuvering the project through the corporate political environment to gaining consensus among
cross-func-tional business users with different objectives. Usually, a data warehouse champion has to spearhead the project to ultimately make the data ware-house successful.
The data warehouse champion is typically a fairly senior business user with a strong understanding of the information technology environment. He must understand the political landscape, have the capability to bring tough issues to a consensus, and should report to a senior corporate sponsor during the data warehouse implementation. Meeting these crite-ria is the best way to ensure that the champion will prove to be a real champion when the chips are down.
The champion must be firmly convinced that a data warehousing so-lution will meet the requirement and solve the defined business problem, to the extent of betting his or her reputation on the implementation. He will also ensure that the right team of professionals is involved in defin-ing the business problem to be solved, and ultimately in developdefin-ing the data warehousing solution that will meet the requirements.
A key element in the champion’s involvement is the requirement to challenge information system specialists, to work with them for the ben-efit of the corporation, and to represent the business users in defining methods of access and presentation of the wide range of information to be derived from the data warehousing system.
Use Detailed Historical Data
The foundation of every successful data warehouse is the detailed histor-ical data on which it is based, and this is the third crithistor-ical success factor. One approach, which has been used by information systems depart-ments to manage the volume and complexity of the issues associated with navigating through weeks, months, or even years of detailed trans-action data, is summary data structures. Although these elements have of-ten become a preferred strategy for implementing decision support systems, they frequently become a detriment to the data warehousing system and its original intent. Summary data structures inevitably fall short of meeting requirements, for several reasons:
• Obscuring data variations: Because they are only summaries of infor-mation, they tend to obscure important data variations, masking im-portant variations in corporate data which may point to problems, indicating areas where successful techniques have been applied in the past and may be applied in the future.
• Complex maintenance: Another deficiency of summary structures is that their maintenance can be fairly complex and quite resource in-tensive, requiring a significant amount of updating to reflect adjust-ments to transaction data.
• Single static scheme: A final deficiency of summary structures is that they are usually created using a single static scheme for organizing transaction level details into a coherent and manageable information format. This limitation ultimately causes the summary structure to fall short because it prevents the business user from viewing the data in a manner conducive to a discovery process. In short, most of today’s business problems or opportunities cannot be identified using a few, limited static views of the business activity.
Summary data tables do have a place in data warehouse design. How-ever, they should not be considered as an alternative to storing detailed data, but rather as a technique for solving some very well-defined perfor-mance problems.
Apply Technology
The fourth critical success factor is that a successful data warehouse im-plementation will apply technology to the business problem. One tech-nology which has been applied to the data warehousing solution is symmetric multi-processor (SMP) computer hardware supporting a rela-tional, multi-dimensional, or hybrid database environment.
More advanced solutions use massive parallel processor hardware (MPP). In a decentralized data warehouse architecture, this solution will probably employ middleware to coordinate wide area access. Further-more, it will entail the use of graphic user interface (GUI) application tools (either developed in-house or purchased off-the-shelf) and online analytical processing (OLAP) tools to present the volumes of data in meaningful formats.
There are a broad range of architectures which can meet the require-ments of a data warehousing system, and the selection of the right technol-ogy is a critical factor. However, architectural issues should only be approached when the business problems to be addressed are clearly un-derstood. The technology should always be applied as part of the solution. Evaluations of different technology can consume significant amounts of time and energy. It is better to work with vendors that can provide ref-erences which relate to an organization’s requirement. Other sources of information are technical publications, seminars and conferences, and re-search organizations that have conducted studies and evaluated a variety of different issues around data warehousing. Knowledgeable individuals in organizations that have implemented a data warehousing solution are also a major and extremely useful information resource.
Trust The Data — History Does Not Lie
The fifth and final critical success factor in a data warehousing imple-mentation is realizing that historical data is a strategic asset, since it is a
source of corporate truths that do not forget or deceive. Human percep-tion and memory can be faulty, and the data warehousing system should not be entrusted to a process which relies on the human memory.
Precise point-in-time readings of key business indicators can help rec-reate a thumbnail sketch of past business events. They can also forecast the success of a future event, potentially reducing the probability of rec-reating previous business disasters.
However, data alone will not solve a business problem. Specialists with specific information system skills will be needed to scrub, load, ac-cess, and present the gigabytes and terabytes of transaction data gener-ated each year by the business. Statisticians and business analysts can interpret the business information distilled from all the detailed data, and provide the business analysis and predictive models that project future business trends.
While history does not lie, it can sometimes mislead. Inconsistencies, incomplete or absent metadata definitions, data loss from media corrup-tion, or unnecessarily restrictive retention cycles, are potential serious threats to the quality of data residing in a data warehouse.
CONCLUSION
There are no guarantees with data warehousing implementation; howev-er, the probability of success will be increased significantly if these five critical success factors are addressed. Therefore, it is important to consid-er these critical factors long before the first quconsid-ery is run or the first gi-gabyte of data is loaded.
A data warehouse system is one of the most complex applications which an organization can implement, since it involves the core business of the organization and a large part of its transaction and business histo-ry. It will have a dramatic impact on each information user. While it is not in itself a solution to a business problem, it provides a means to a solu-tion, one which will involve the entire organization in a major cultural change. This change will enable employees to use detailed information as a key to knowledgeable corporate decision-making.
Duane E. Sharp is president of SharpTech Associates, a Canadian company specializing in the communication of technology. An electronic engineer with more than 25 years of experience in the IT field, he has authored numer-ous articles on technology and a textbook on interactive computer terminals, and he has chaired sessions at Com-dex Canada. He can be reached at [email protected].
