Mainframe Performance Management: A New Twist
Produced by SearchDataCenter.com
Presenter: Wayne Kernochan
Sponsored by
Copyright © 2008 CA. All Rights Reserved. Reproduction, adaptation, or translation without prior written permission is prohibited, except as allowed under the copyright laws.
Mainframe Performance Management: A New Twist
This document is based on a CA/TechTarget Webcast entitled “Mainframe Performance Management: A New Twist.”Matt Stansberry: Hello and welcome to the SearchDataCenter.com Webcast entitled “Mainframe Performance Management: A New Twist.” My name is Matt Stansberry and I will be your moderator. Joining me today is our guest speaker, Wayne Kernochan. Kernochan is currently President of Infrastructure Associates. He has been an industry analyst for many years and previously worked with Aberdeen and Yankee Group where he focused specifically on mainframe software. He is also a long time contributor to TechTarget's Mainframe and DB2 expert content. Thanks for joining us, Wayne.
Wayne Kernochan: Thank you very much Matt. I am going to be talking today about performance management software specifically as it relates to the mainframe. In my coverage of infrastructure software problems and solutions, I am finding that this is a topic of increasing importance to IT. I will get into exactly why that is as we move on further in the presentation. Before I do that, I would like to talk about what performance management means to me, what the definition is, and the reasons for doing it. Recently there has been another definition of performance management which is gaining popularity and that's in the business intelligence area. In this area, performance management means providing software to the business users: the CEO, the CFOs, and the top layer of management. This allows you to manage the performance of your organization via dashboards and so on. That's not what I am talking about today, however. Nor am I talking about disaster recovery. Instead, I am talking about the more traditional
meaning of performance management. Here we are really talking about the side of the administration of systems and applications that focuses on keeping things up as much as possible, excluding reliability, availability, and serviceability (RAS). We are really talking about the part that focuses on delivering decent performance for the applications, for the systems, and for the response time to the end user.
T
he key point about this is that performance management involves bringing the every last ounce of performance out of an application. Typically, given the types of applications that I see out there, most people are really talking about achieving “good enough.” When I say “satisficing,” it’s a term for saying that it is not possible to optimize everything so what you did was just to make sure that all the applications are good enough. So again, the focus here is little different. It's on meeting service level agreements without feeling like you need to go way beyond them. It is also about the long term because your systems are constantly in motion. They are constantly growing in most cases. So what you really need is aperformance management solution that scales so that you can continue to achieve that adequate response time. Typically in the cases I see, it takes two seconds or five seconds for customer facing applications. So again it’s a different flavor. When you look at Mainframe performance management, it’s yet again a slightly different flavor because the Mainframe has its own strengths in virtualization and load
Mainframe Performance Management: A New Twist 2 balancing. We need to look at load balancing in particular because, as we are seeing now, you can cram a goodly number of applications and virtual machines onto a mainframe. Some users are even reporting around 200 applications and virtual machines. In that case, that’s a whole different level of load balancing compared with traditional performance management.
All right, so what’s the big deal? Well, the flip answer is that if you look at IBM’s latest quarterly report you will see that sales of mainframes are up 32%. This is phenomenal when you think about it because this is a product that's been around for donkey’s years. What’s going on? Why should the mainframes be so important? I have been monitoring the last three years of marketing messages as this type of sale has begun to ramp up. What I am seeing is that frequently the core of the value propositions for the IT user is total cost of ownership (TCO), and specifically per application TCO. Yes, other issues like security are also important, but what tips the balance toward putting larger work loads on mainframes is the ability to achieve a better per application TCO.
Depending on which study you look at, the tipping point has tended to be about 10 to 20 applications. Above that level, it’s likely that the TCO for the mainframe is less than that of Unix servers, multiple and distributed or not, and less than PC servers. The reason this relates to performance management is due to the components of that TCO. If you look at the figures in one of the available IBM studies, and you dig down deep into it, it comes as no surprise that hardware costs are even on a per application basis. This is true even when you are talking about more than 20 applications. The costs are still going to be greater with multiple PC servers or multiple Unix servers.
What about software costs? Well, as it happens, right now you can save quite a bit on software license costs by putting them all on the same machine. This is because of the way that software vendors typically handle their licensing costs. Now that’s a short-term aspect because you can’t guarantee that vendors won’t change their licensing costs in the future. So that may or may not continue. But while it continues, it tends to balance out the added hardware cost. So the real key, the long-term key, to why the mainframe has become so attractive is the personnel cost. Staff expenses constitute most of the administrative cost. That is tied to the fact that you are running everything on one machine. Physically you don't have to worry about administering a whole bunch of extra communication wires and so on. You don’t have to bound from machine to machine or worry about machines in different geographical locations. You are really dealing with only one machine and that has a major effect on personnel costs. Another reason for this is that the mainframe is well architected to be able to cram lots of virtual machines onto it. That's a function of the performance characteristics of the application. If you really want to drive the TCO down on the mainframe, then you want to add more applications. In order to add more applications, then you have to establish capacity on demand. That’s not so complicated; it's just a “screening issue.” A screening issue refers to getting as much performance as you can out of the applications that you have and effectively load balancing those applications.
That gets back into performance management. Performance management really gets at the core reason that folks are looking to the mainframe. The more effective your performance management is, the better
your short and long term TCO advantage. These mainframes are being run at 80% of capacity and higher. The more you can squeeze out of them, the better.
So why hasn't performance management been a big topic in either the mainframe area or the general platform area up to now? There may be cultural resistance or other reasons why folks think performance management is nice but not necessary. According to many administrators, the focus has been on disaster recovery, including governance and downtime. Obviously that has a major effect on business. It’s the clear target. Performance management is nice if we can get to it. I think there is also a more subtle effect going on in the performance management area, though, and that has to do with software development. What I have typically seen in software development is that performance testing and performance management are afterthoughts. Yet if you are really doing the development well, and if you are really intending to keep the quality of the application high, performance management is critical. But it has not been part of the reflex action of the development organizations and the administrators interfacing with them.
Early this year I was reading the Boston Globe. There was a nice letter from somebody to the career advisor that said “Dear Sir, I recently worked for a major software vendor as a performance and stress tester. We discovered that the big product that they were about to come out with was slow as a dog and consumers wouldn’t accept it because of its performance. When we informed the manager of the product of this, his reaction was to go to his boss and say, ‘You know, performance testing really isn’t necessary anyway and we need to save costs, so please let them all go.’ So I got laid off and sure enough the product bombed in the market. Is there anything I can do?” Of course, the answer was “tough luck” because that was the attitude towards performance testing.
Mainframe Performance Management: A New Twist 4 It is also worth noting what mainframe performance involves, what it used to be, and what we are moving toward. In the past, workloads were mostly z/OS, CDM, and also DOS for that matter. It was all about optimizing one particular machine, but not necessarily too many virtual machines. Sometimes it was just one or a few applications. It was really, even more than in the distributed world, about focusing on non-communicating, non-distributed processes. IT shops typically reported that whatever tools they got for performance management had to be automatically turned on; if they weren’t, they didn’t bother to do it. They expected things to work out-of-the-box. It was not a high priority to understand how to tweak these capabilities. There is a lovely example from the mid-nineties where basically Tandem was able to go into an IBM customer and announce that they had achieved better performance than IBM. IBM went in afterwards and looked at that customer. They discovered that the reason Tandem was able to beat them in performance was because Tandem had carefully optimized the IBM system for their own application. The customer had just taken the IBM system and slapped it in. He didn’t bother to do the performance speed up.
I still see that type of thing going on in the mainframe environment. It is something to think about as you work toward performance management on a mainframe. You can probably get a feel from what I am saying about how things have changed. Today mainframe performance management is seen as a valuable effort that can accomplish a great deal. Obviously, based on whatever reports you look at, a surprising amount of the new sales of Z10 and the usage of mainframes has to do with Linux and UNIX workloads, especially Linux. Many of these applications offer server consolidations. Those are the cases where you see 150 to 200 Linux applications put on the mainframe, with or without business critical z/OS applications.
To a certain extent z/OS applications have increased the numbers of web users. But the new Linux applications are taking us well beyond that. They are basically offering distributed communications, perhaps between the applications or out on the web. In terms of communications with clients, you all know that the typical PC is not being screened very often out on the web. That in turn means that if you really want to manage performance much more than ever before, you are also dealing with performance outside the mainframe. Yes, while virtual organization will take you very far, you still meet end-to-end. Obviously, the increased virtualization allows you to do some really nice things related to performance management. These have to do substantially with what you put on the mainframe. Of course, you can communicate one heck of a lot faster between applications in two separate virtual machines. You can also over align, which may be geographically distributed across the web. Performance management allows you to start shifting things around to achieve added performance as effectively as possible. Finally, as I touched on before, users are cramming as many applications as possible onto the mainframe for maximum TCO and getting reasonable performances out of them. But it is important to make sure that you load balance them effectively so that you can achieve a certain response time and minimize delays.
It is increasingly important that you get effective software to do mainframe performance management. Obviously it should be end-to-end. The vendor should be experienced in mainframe and Linux
administration. But also, these new environments aren't only end-to-end. The addition of the web has made the software stacks deeper and the points of vulnerability for performance management more numerous. Now you not only have to deal with the applications, such as the database, but also the middleware, such as the web server, the application server, the enterprise servers and the
communication software. These can have a significant negative effect on performance, so your performance management software has to be able to look at those aspects as well.
And finally, it is good to have a highly automated system. I think everybody I have talked to agrees that people who are really experts in the mainframe environment and are engaging in this type of performance management as part of their administrative tasks are all about 65 years old. Replacing them is not always straightforward, even in a case like this, because some of the applications will continue to be z/OS. Consequently, the software should have a high degree of automation and standardization across the mainframe, at least in its interfaces across the mainframe and the Unix/Linux shots.
There are some other more tentative conclusions that make mainframe performance management effective above and beyond the software. In the past I have found that the effect of database
performance, especially long term, on TCO is often underestimated. A favorite example of mine concerns people who continue to treat reorganization of the databases as an afterthought. In point of fact, if you don't reorganize your databases, you will see a 10% degradation in performance per year. This is
especially true if it's a database or an application involving lots of updates. This is because every time you delete or insert information, you stick the data in a different place. You stick a pointer from the original place to the new data location. Pretty soon instead of one I/O, you are chasing down chains of I/O. The fix is to run a reorganization. Doing this even just once a year will restore the data in a most efficient manner, yet people continue to fail to emphasize that.
Mainframe Performance Management: A New Twist 6 Because the database is an important aspect of performance, zIIP processors will allow you to wring more performance out of a key aspect of the overall application performance. Also, I find that more and more IT organizations look at simply being able to pass data back from the operational performance management software to the development organization. In other words, they don’t just care that this error crashed the system but also that this problem is really slowing the system down. So some sort of formal or informal system for passing that type of operational data back to the developers is really effective. It's especially effective because we are actually seeing that folks are treating mainframes as robust enough to do this. They are becoming comfortable putting their testing and development environments as virtual machines on a mainframe. Third, there is a lot of tweaking that you can do which basically shifts over processing power to the PC client out there somewhere, or shifts it back to the mainframe.
There is a common conception that interactions over the web involve clients approaching the old green screen dumb terminals that have been the traditional mainframe staple. But with the advent of rich clients you can absolutely achieve better performance by improving the balance of responsibilities between a mainframe and a client. A bit of attention to that can have surprising effects. One more trend is likely to have an increasingly important role over the next one to two years in the area of mainframe performance management. As the new workloads arrive on the mainframe, the differentiation between mainframe platforms and Unix/Linux platforms is obviously decreasing. People are going to begin to think in different terms about what goes on the mainframe and what goes on the Unix server. The net result of that will not be to say “Let’s put all applications on the one or all applications on the other.” There are things that the mainframe does better, such as security, and there are other things that UNIX and distributed systems do better. What you are much more likely to have is a type of arrangement where the mainframe is the hub.
It will be used for most but not all of the applications for a particular task, like achieving greater security or robustness, and so on. In that case, performance management has to think about how it works across both mainframes and UNIX/Linux servers and determine recommendations about where the margin should be between the two. The other trend that I see coming in the next one to two years is that, increasingly, performance management must provide the kind of interfaces that are familiar with UNIX/Linux but not yet familiar with mainframes. And things will need to be standardized across both platforms because mainframe administrators are becoming older and grayer.
Another issue that everyone is talking about these days is Green IT. The focus so far has been especially on the hardware and designing facilities. Yet as people have pointed out, there is a significant role for efficient software applications to play in energy savings by allowing you to turn off certain machines even during peak loads because their performance management has made them efficient. So I expect that performance management software will begin to look at potential effects not just on performance but also on energy consumption, or at least to allow you to factor in that type of consideration.
In sum, based on my research and observations, mainframes are once again becoming important to business. Performance management and the combination of the two are also growing in importance. In some cases, such as key online bank batch applications, systems can run 500% faster. This provides a better costumer experience for a telecom company, or much greater customer satisfaction as measured by an insurance company. These are the types of cases where I see the use of mainframes and
performance management becoming increasingly important. Consequently, you will need good
performance management software and practices. As you look to the future, you should look for advance solutions, mainframe experience solutions, and deep solutions as well as more automation in the
performance management software. As a final point, the trends that I have cited here are likely to speed up over the next one to two years. I see nothing on the horizon that indicates a slowdown.
Matt Stansberry: Thank you, Wayne, for an excellent presentation. For more information and to read more of Wayne’s columns, check TechTarget’s Wednesday Enterprise Systems Updates Newsletter.