O r a c l e 11 i D a t a U p g r a d e R o a d m a p —
A L i f e c y c l e - b a s e d A p p r o a c h
Grant Gasson,
Director of Financial Systems
Apollo Group, Inc.
An upgrade to Oracle 11i puts a premium on an organization’s ability to manage its data. This white paper will explain how the Apollo Group, Inc. used a proactive application data management approach with online data archiving to address the complications and complexities of an Oracle 11i upgrade. The paper will explain how a lifecycle-based approach enabled us to achieve our downtime and performance objectives before, during, and after the upgrade, including the need to implement data archiving as part of ongoing operations. And, finally, the paper will highlight the lessons learned during this experience.
During our recent upgrade from 10.7 to 11i, the Apollo Group demonstrated how a proactive application data management practice that includes data archiving can mean the difference between meeting our upgrade objective or suffering an unacceptable break in business operations. Using the LiveArchive solution from OuterBay Technologies, we were able to archive data from our Oracle GL and AR modules. The resulting 22-percent reduction in the size of our production database enabled us to slash our upgrade outage window by 12 to 24 hours and accelerate the performance of a key report.
G o o d E d u c a t i o n i s G o o d B u s i n e s s
Apollo Group Inc. was founded in 1973 to provide a source of continuing education for working adults. The company’s founder, John Sperling, believed that, for most adults, the old days of lifelong employment with a single employer would be replaced by lifelong learning and employment and multiple employers. This anticipated shift soon materialized and Apollo quickly established itself as a leader in the higher education industry.
Apollo grew rapidly, posting impressive enrollment and financial results. Between 1997 and 2001, student enrollment and revenues for its University
1
of Phoenix, Institute for Professional Development, College for Financial Planning, and Western International University subsidiaries more than doubled. Today, Apollo now serves nearly 150,000 students.
This fast paced growth demanded constant effort from our IT staff to support that growth. Because of the growing footprint of the organization, we needed our technical systems to accommodate more customers, more capabilities, more locations, and more data.
Apollo first implemented Oracle Financial Applications in 1989 to address some of these growth-driven issues. In the 12 years since then, we’ve completed at least eight upgrade cycles to implement new functionality and technologies necessary to keep pace with data growth.
Our internal IT staff managed each of the upgrades. The resulting hands-on experience has given us a great understanding of the challenges and complexities generated by the Oracle upgrade process. We know what we’re getting ourselves into because we’ve been there and done that. For instance, we’ve learned there are at least two overarching themes to expect from every Oracle upgrade. First, an Oracle upgrade project is a long-term project— usually six to eight months—that requires four to six test cycles and
considerable planning before, during, and after the upgrade. Second, despite these efforts and the best-laid plans, Oracle upgrades tend to handle data inconsistently.
H i g h e r E d u c a t i o n a n d t h e
D a t a R e t e n t i o n C h a l l e n g e
As a provider of higher education, we are subject to stringent state and federal regulations. At the state level, Apollo is required to retain all documentation from every year of operations in that state. With a growing enrollment in 37 states, these data retention policies are responsible for generating significant data growth.
On the federal side, we are subject to Department of Education regulations, especially with regards to tuition refunds. The federal agency requires us to keep clear records of all refund transactions. We don’t want to put the university at risk for non-compliance so we’ve established data retention policies that maintain access to all historical transactions records.
This dual demand for data retention and access had caused our production databases to grow to the point where the database size was slowing performance and causing reports to run noticeably slower. We run more than 10,000 concurrent programs per day so any performance drop, even by a few seconds, can lead to an unacceptable loss in productivity.
Given our data retention and data archive access needs, an effective data archiving solution for Apollo and the University needed to perform several functions.
It needed to correctly identify which data to relocate to the archive. It must then correctly relocate the identified data and do so while maintaining the integrity and formatting of the data.
To eliminate the need for retraining users, the solution also needed to retain access to the archived information through existing Oracle Forms and Reports.
And the solution had to perform all these functions without violating the Oracle warranty or certification.
A p o l l o ’ s U p g r a d e O b j e c t i v e s
In late 2001, we decided to upgrade from Oracle 10.7 to Oracle 11i in anticipation of Oracle’s initial de-support date of June 30, 2002. We had three key objectives for the upgrade. We wanted to minimize the outage window. We wanted to maintain access to archived data. And we wanted to improve post-upgrade performance.
3
O u t e r B a y O A U G F a l l 2 0 0 2
• • •
O u t a g e W i n d o w
We hoped to complete the upgrade over the July 4th weekend to minimize the impact of the outage window on business operations. If the system was down even for a few hours during business hours, it would impact our customer service and cost us up to $50,000 per day in lost productivity. Unfortunately, the growing size of our production database threatened to bloat the outage window and disrupt operations during the following week.
D a t a A c c e s s
Because of the state and federal regulations discussed earlier, data retention is a huge issue for the company. We needed to make sure that all transactions from before the upgrade could be retained and accessed even after the upgrade to Oracle 11i. In addition, our typical student takes classes over a four- to five-year period. Because we planned to archive all AR records older than three years, we needed to provide easy access to archived information related to student billing and payment information.
P o s t - U p g r a d e P e r f o r m a n c e
Beyond the need to address the coming de-support date, Apollo hoped to see a boost in system performance after the upgrade to Oracle 11i. From several forums, including OAUG papers and seminars, we knew that Oracle 11i suffered from an accelerated data growth rate. While the new capabilities promised some key performance advances, this data growth had the potential to choke off some of these benefits. Before we started our upgrade, our existing system was already suffering from a drop in performance. The accelerated growth potential of Oracle 11i threatened to add additional momentum to this decline.
B e f o r e t h e U p g r a d e — T e s t i n g
We started pre-upgrade testing in February of 2002. Initially, we ran our tests on two instances of more than 200 gigabytes each. After only two tests, the database administrator determined that, using the then-current production database, the projected outage window would exceed 88 hours. If so, the system downtime would extend into the following week and create an unacceptable outage during business hours.
To bring the outage window down to an acceptable length of time, we needed to shrink the size of the production database. But we had to do so while retaining access to historical records for state and federal regulations. From past experience, it was clear that Oracle’s purges alone were not up to the task.
Because we were already in the upgrade test process, we needed to find a solution quickly. During the previous year, our database administrator had investigated a data archiving solution from OuterBay Technologies, called LiveArchive. He’d performed a proof concept on LiveArchive and felt the solution provided a proven approach to meet our data retention, Oracle certification, and data management requirements so we moved swiftly. Within a week, we’d contacted OuterBay, signed a contract, and started the
implementation.
B e f o r e t h e U p g r a d e — F i r s t A r c h i v e
The LiveArchive implementation took only four weeks. Installation and testing for GL and AR started in mid-March of 2002 and went live on April 18th. For the first GL purge and archive, we kept a rolling, five years plus the current year history of GL transactions on the production server and purged closed GL transactions from 1988 to 1995. This initial GL purge and archive was completed in three days on April 21. By May 19, we purged and archived all closed, or inactive, AR transactions older than three years from 1994 to February of 1999. And six days later, we completed our first archive build.
5
The results of the data archiving were dramatic. The size of the database was trimmed by 22 percent. Based on our tests, we expected this reduction to minimize the outage window and make it possible for us to complete the upgrade over the targeted weekend.
The LiveArchive solution also addressed our data access and retention objectives. Historical data that might be needed for regulatory compliance was not deleted; rather it was relocated to an easily accessed archive server. In addition, the records remained intact so they could be accessed using our existing Oracle Forms and Reports.
Though pre-upgrade results are no guarantee of post-upgrade performance, we did see evidence that LiveArchive would deliver improved post-upgrade system performance. For instance, we often run a report that merges historical and archived data. Before LiveArchive, this report took approximately two hours to run. After archiving, the report ran substantially faster and was completed in only 12 minutes, a 90-percent reduction in runtime.
D u r i n g t h e U p g r a d e — t h e O u t a g e W i n d o w
The potential outage window of 88 hours was the primary catalyst for our decision to implement a data archiving strategy. We simply could not take the risk of letting the upgrade disrupt our regular business hours. Our archiving strategy virtually eliminated this risk. After LiveArchive reduced the production database by 22 percent, the outage window was slashed by at least 12 hours. We believe the actual savings was closer to 24 hours. In a time-critical environment where unproductive hours can harm customer satisfaction and cost tens of thousands of dollars, these savings meant we were able to keep our systems open for business.
During October of 2002, we plan to upgrade all archived transactions to Oracle 11i. One of the benefits of LiveArchive is its ability to apply all Oracle upgrades and patches to the archive. This approach enables us to keep our
production and archive servers on the same release which, in turn, eliminates the need for additional applications or training to access the archive.
P o s t - U p g r a d e — P r o a c t i v e A p p l i c a t i o n
D a t a A r c h i v i n g P r a c t i c e
Though we’re still completing aspects of the upgrade, we have instituted an ongoing application data management practice that incorporates LiveArchive to improve the post-upgrade performance of our Oracle 11i system. Our current plan is to add to the archive guidelines we established prior to the upgrade. Every quarter, we’ll continue to archive AR records older than three years. In addition, we’ll include PO and Payable records older than three years. At the end of each fiscal year, we will archive all fiscal years older than five years.
This post-upgrade data archiving strategy has two primary objectives. First, we want to keep system performance consistent even while student enrollment and accompanying transaction data continue to grow. Keeping report runtimes short is critical to our productivity and customer service. We currently run 10,000 concurrent jobs everyday. If poor system performance adds only two to three seconds in runtime to each of those jobs, the aggregate slowdown will include five and a half to more than eight hours of additional processing time every day. That’s similar to adding an extra day of processing every third day, an unacceptable situation.
Secondly, a long-term data archiving strategy can help us prolong the life of our existing technical infrastructure. We currently run Oracle 11i on a Sun E4500 with 12 gigabytes of memory and 12 300MHz processors. Typically, accelerated data growth requires more storage and processing capacity. Using LiveArchive we’re hoping to cap those demands and extend the life of our Sun E4500 platform to its five-year depreciation schedule.
7
L e s s o n s L e a r n e d
After performing eight Oracle Applications upgrades in 12 years, we know that every upgrade is a learning experience. This upgrade was no exception. Our biggest lesson was to discover the importance of LiveArchive. Our initial upgrade plans didn’t include plans for data archiving. When it became apparent that we needed a solution to salvage our upgrade objectives, we had to scramble to put a strategy in place. Luckily, our DBA had already done plenty of legwork so we were able to move forward quickly.
Given a second chance, we would have made LiveArchive an integral part of our upgrade plans and performed the archive earlier in the process. With a little preplanning, we could have put more thought into our archiving guidelines and used the upgrade tests as a truer indication of the expected outage window. For instance, as stated earlier, we established three- and five-year thresholds for our AP and GL records. If we had the extra time to change these guidelines, we would have made the thresholds consistent for all records.
C o n c l u s i o n
The Oracle 11i upgrade process is a complex and challenging undertaking. From our most recent experience, we learned firsthand how a proactive application data management practice, that includes OuterBay LiveArchive, can shrink the size of the production database and test instances. As a result, we were able to accelerate the upgrade process and cut the length of the outage window. These time-savings were essential given the importance we placed on eliminating any lost business days due to the upgrade outage. The ongoing legacy of a more efficient production database due to data management and data archiving will also extend into our post-upgrade operations. We’ve already seen evidence that LiveArchive will enable our Oracle 11i upgrade to deliver faster reporting, more effective data retention, easy access to historical data, and higher system performance.
OuterBay LiveArchive proved to be an invaluable solution for managing application data before and during our upgrade efforts. And, based on what we’ve seen already, we plan to incorporate its data management and data archiving capabilities as an ongoing practice in our day-to-day operations and an integral part of subsequent upgrades.
9
C a m p b e l l , C A 9 5 0 0 8 U . S . A Te l 4 0 8 . 3 4 0 . 1 2 0 0 F a x 4 0 8 . 3 4 0 . 1 2 9 9 w w w. o u t e r b a y. c o m e - m a i l i n f o @ o u t e r b a y. c o m
© 2002 OuterBay Technologies, Inc. All rights reserved. OuterBay, Application Resource Management, Application Resource Manager, LiveArchive, and Instance Generator are trademarks of OuterBay Technologies, Inc. Other trademarks
