1
Up-Front Design Versus Evolutionary Design in Denali’s
Persistence Layer
Jim
Little
Titanium I.T. LLC
3062 SW 153
rdDr.
Beaverton, OR 97006 USA
+1 503 605 5442
[email protected]
ABSTRACTThis experience report describes the evolution of a persistence architecture over the course of nine months. The persistence architecture was developed as part of a common code base for internally developed web applications. The code base, named Denali, was developed from the ground-up using Extreme Programming and a team of six programmers. Denali was originally developed using up-front design and then extended using evolutionary design. Evolutionary design was found to yield better results than up-front design.
Keywords
Extreme Programming, persistence, evolutionary design
1 INTRODUCTION
One of the controversial tenants of Extreme Programming (XP) is that evolutionary design can replace up-front design. In XP, the programmer is encouraged not to design his code in advance, but to use tests and refactoring to drive the design [1]. This approach has been greeted with skepticism on many occasions (e.g., [2]). Is it really possible for a non-trivial system to evolve a consistent, maintainable architecture using nothing but evolutionary design?
In this paper, I present my experiences with evolutionary design in a medium-sized project. Although I was a member of a team of developers, this paper is a personal account of my experiences on the project. As a result, I’ve written the paper in the first-person perspective.
2 THE PROJECT
In May 2000, I was hired to help a team of programmers reduce their maintenance costs. Their manager told me that a new person would arrive and be productive for a few months, but then would come to a standstill, spending all their time updating existing software. After reviewing the situation, I discovered that there was massive duplication of code, no shared knowledge about projects, and no clear prioritization of needs. I recommended that we switch to XP to resolve the prioritization and knowledge sharing issues, and that we develop a common code base for all applications to resolve the maintenance issues.
As of June 2001, the project has been in progress for 13 months. We have been using XP from the beginning to
develop all of the new code. The project has been an unqualified success. Although the customer would prefer to have more software developed more quickly, we have clearly met her goals. New features are delivered regularly, knowledge is spread throughout the team, and features are clearly prioritized. The common code base, named Denali, has worked out as well. There are currently 18 web-based applications in Denali, all using a common object model and persistence framework.
Denali Facts
Project Length 13+ months (still active) People 4-12 programmers; currently 5 Programming
Language
Java servlets (no Enterprise Java Beans or Java Server Pages)
Tests 1586 total, 91 acceptance
Production Test Total
Classes 405 368 773
Methods 3,376 3,245 6,621
Statements 10,342 14,254 24,596
Denali was developed from the ground-up using XP. From the beginning, the most complicated part of the system has been the persistence architecture. This paper describes the changes that have occurred in the persistence architecture as we applied XP to the problem.
3 THE PLAYERS
I was the technical lead and XP coach for Denali. When the project started, I had about four years of professional programming experience. Of that time, only the prior six months had been spent as a technical lead, and only the prior three months had been spent practicing full XP. I had limited experience with relational databases and no prior experience with object-relational persistence. However, I was an expert Java programmer and had a solid grounding in the theory and practice of object-oriented programming. The remaining developers had more years of programming experience, but limited exposure to programming methodologies and techniques. For most of them, Java and
object-oriented programming were new technologies.
Did we really practice XP?
Planning Game Yes
Frequent Releases Yes
Simple Design Yes
Unit Tests and Acceptance Tests
Yes
Continuous Integration Yes
Coding Standards Yes
Refactor Mercilessly Yes
Pair Programming Yes
Collective Code Ownership Yes
40-hour Week Yes
System Metaphor Limited On-site Customer Limited
4 TAKE ONE: UP-FRONT DESIGN
When we started Denali, I believed that it was destined to become a large system. At the time, I knew of XP’s insistence on evolutionary design, but I had no direct experience with it. As a result, I hedged my bets: I researched potential architectures for several days and sketched out a design in advance. The result was a five-layer architecture based on Martin Fowlers’ Analysis Patterns [3]. My pairing partner felt that the design was too complicated, but I “knew” we were going to need to manage complexity in this project and believed that the additional up-front complexity would reduce problems down the road. Was I ever wrong!
The first approach consisted of five layers. (See figure 1.) Business objects resided in the middle layer and modeled the domain. At either edge resided the presentation and database layers, each protected from the business layer by a translation layer. The presentation layer would be for applications. Each application would have its own presentation and a façade to protect it from the expected complexities of the business layer. At the other end, the database layer would model the database. The persistence layer would translate the relational data in the database into objects in the business layer.
How it worked
Although the above architecture looked fine on paper, it encountered some problems in practice. In order to persist an object’s data, the persistence layer needed full access to the business objects’ instance variables. I didn’t want to provide accessors and mutators for every single instance variable, as that would violate encapsulation. And I didn’t want to make the variables public for the same reason. Eventually, I came up with the idea of using Java’s inner classes to provide back-door access into the business objects (see figure 2).
In Java, an inner class has an implicit reference to the variables of the outer class [4]. Because the inner class is within the scope of the outer class, it also has access to all of the outer class’s private member variables. By providing public methods on the inner class, the programmer can provide additional interfaces to the object’s data. In Denali, we called these inner classes “Persistence Friends,” after the “friend” access level of C++. Denali’s Persistence Friends allowed the persistence layer to act like C++ “friends” and access the private instance data of business objects. Because the Persistence Friends were actual objects, we could control access by limiting who received an instance of a friend. It seemed like a great solution.
Reality sets in
Unfortunately, the idea turned out to be extremely complex in practice. The local development team, with no prior experience in Java or object-oriented programming, was completely lost. Furthermore, the approach required a large amount of code to accomplish anything significant. After using persistence friends for several months, the other developers presented me with an ultimatum: Get rid of the persistence friends!
Conclusion: Up front design
My first attempt at a persistence architecture was driven by up-front design. I was able to come up with a workable solution, but one that was far too complex. In retrospect, I should have introduced the layers much later, when there was a demonstrated need. I believe this approach would have reduced our costs in the short term and would have resulted in a better design in the long term.
From this point onward, architectural changes were driven by XP principles, not up-front design. Every future change was a change for the better.
5 TAKE TWO: PUBLIC VARIABLES
The near-lynching inspired by the persistence friend architecture caused me to rethink my attitude towards encapsulation. The entire purpose of the persistence friends was to protect business objects’ instance variables from untoward manipulation. But who exactly was I protecting the code from?
Operating on the assumption that we could trust ourselves, the team established the rule that business objects’ instance variables could be public and accessed by tests and the persistence layer, but not by anything else. This simplified the code immensely. Now the persistence layer just accessed and modified instance variables when it needed to (see figure 3).
The team didn’t have any problems following the public variable rule. As time went on, though, our tests became too closely coupled with our business objects. After a few months, we started having problems refactoring our code.
Conclusion: Public Variables
The move from persistence friends to public variables was a huge win. Some people might see it as a step backwards: we went from a solution with very well defined access control to one with no access control at all. The public variable solution was superior, though, because it was far
simpler. But it suffered from close coupling in the tests and still had problems with cleanly separating the persistence and business layers.
6 TAKE THREE: SCHEMA ABSTRACTION
One thing I’ve noticed about our XP project is that we’ve never been completely happy with our design. We struggle to get acceptance tests working, and then the architecture feels wrong. We work to get the architecture right, and then the tests need work. We fix the tests, and then we have mock objects [5] scattered all over the place.
I don’t see this as a problem with our project or XP, though. Instead, I think it’s our biggest strength. We’re never satisfied with our design and are constantly evaluating our system to see what could improve. There’s always some long-term scheme to improve this or that part of the system. Once those changes have been made (often over the course of a month or two), new problems or ideas have come to light.
This was the case with our database schema abstraction. We had been using procedural helper methods for a lot of our test set up. We knew that the procedural nature of these classes was a code smell [6], and once we fixed the persistence friend problem, we had a chance to work on them. Most of the classes set up and tore down records in the database, so we refactored them to represent database rows. One instance represented one row in a single database table.
This approach was so successful that we decided to migrate it to our production code. Up until this point, our database abstraction had consisted of a thin layer over Java’s Database Connectivity (JDBC) package. All of our persistence classes had hand-coded SQL in them.
We established the basic Row and Table abstraction fairly quickly. As time went by, we identified many things that could be abstracted out into parent classes. Now all of our basic operations, such as insert, update, delete, and lookup by primary key, are implemented in the parent class and are automatically available for every table.
Conclusion: Schema Abstraction
The introduction of Row and Table classes was a significant improvement to our architecture. With this
change, we were able to move all of our SQL out of the persistence layer and into the database layer. We removed duplicate queries and factored many common needs into a shared parent class. Most importantly, our SQL was now grouped according to database table, not business object. This important change made our code much easier to understand.
7 TAKE FOUR: FORKLIFTS
The introduction of Row and Table classes as an abstraction for our database schema was a huge step forward, but we still had a major problem with our architecture: the persistence layer and the business layer were far too closely intertwined. The persistence objects were merely an intermediary between the business layer and the database. When called, they asked the schema abstraction for a variable or set of variables and plugged that back into the business object via the public instance variable backdoor. With the introduction of the Table and Row schema abstraction, all of their SQL was moved and they became nearly useless.
At the same time, the business objects were far too complex. The business rules for our system were trivial, but the business objects were large and difficult to understand. The problem was that the business objects managed when the persistence objects were called. They had to keep track of which data had been loaded already and only ask the persistence layer for new data when it hadn’t already been pulled in.
The schema abstraction gave us the opportunity to reexamine the relationship between our business layer and the persistence layer. With the new abstraction, we saw architectural changes that hadn’t been clear before.
There were two problems with the existing architecture. First, the need for the persistence layer to access the business layer’s instance variables was a perpetual problem. Second, the business layer shouldn’t have been responsible for managing caching.
With the schema abstraction in place, a new direction was clear: We would move all of our data into the persistence layer and have the business objects focus on business rules. Each business object would contain only a single corresponding persistence object. The persistence object would store all the data for the business object. We
expanded our system metaphor to handle this new approach: The database was the “warehouse” where our data was stored, and the persistence objects were “forklifts” that carried around our business’s data (see figure 5).
Conclusion: Forklifts
The forklift version of our persistence architecture has been in place for seven or eight months. It’s worked great for us so far. For the first time, we’re happy with our persistence architecture. Although we continue to make minor improvements, I don’t think we’ll see any more significant changes to this part of the system.
Overall, the forklift approach has worked wonderfully. The code is simpler. The time at which objects are loaded and how we deal with missing information is clear for the first time. I can’t stress how much of an improvement this approach is over the previous way of doing things.
9 CONCLUSION
Can evolutionary design replace up-front design? In our case, absolutely. Up-front design yielded a workable but overly complex design. Evolutionary design yielded improvement after improvement. It’s been an immense success.
The evolutionary approach has yielded two surprising results. First, it's easy. Coming up with design improvements required no real effort, because they gradually bubbled up from my subconscious over the course of months of working within the system.
The other surprise was how effective evolutionary design is. I believe that the current design is far superior to anything I could have come up with in advance. The reason appears to be that, with evolutionary design, I could simply see farther. As we made each change, new approaches that were previously hidden became visible, and even obvious.
I’ve noticed one major drawback to evolutionary design. Architectural refactorings are hard and take a lot of work. When we moved from one persistence approach to another, it could take several days per business object. In our case, we only migrated business objects that we worked on, but
that has resulted in some infrequently-used objects in our system using old approaches months after a new approach was identified.
Overall, the evolutionary approach has been extremely effective. I would recommend it as the primary approach to architecture and design for any project in which aggressive refactoring is feasible. As a result of evolutionary design, our system is a joy to work with.
ACKNOWLEDGEMENTS
This paper would not have been possible without the support and patience of the Denali programming team. Thanks to David Bean, Lance Black, Ryan Davis, Jarid Love, and Jared Whitlock for their willingness to try new things. Thanks especially to Kim Eaves for letting us turn her traditional procedures upside down.
REFERENCES
1. Beck, K. Extreme Programming Explained, 2000, Reading, MA, Addison Wesley Longman, Inc., 107-108.
2. Evolutionary Architecture discussion on Ward Cunningham’s Wiki-Wiki Web. On-line at http://c2.com/cgi/wiki?EvolutionaryArchitecture.
3. Fowler, M. Analysis Patterns: Reusable Object Models, 1997, Menlo Park, CA, Addison Wesley Longman, Inc., 239-256.
4. Gosling, J., et. al. The Java Language Specification. Online at http://java.sun.com/docs/books/jls/index.html. 5. Mackinnon, T., Freeman, S., & Craig, P. Endo-Testing:
Unit Testing with Mock Objects. On-line at http://www.sidewize.com/company/mockobjects.pdf. 6. Fowler, M., et. al. Refactoring: Improving the Design
of Existing Code, 1999, Reading, MA, Addison Wesley Longman, Inc., 75-88.
