The free
Performance Management
Solution
...because performance matters!
inspectIT
Happy Ending?
After two years of developing an improved version of the internal business system based on the newest technology and integrating the gathered requirements of the emplo-yees, the development lead was happy to announce: “The application was deployed successfully and starting this second we are live. We have done it, folks, the hard part is over”. After the last few months of working long hours to keep the fixed deadline, the development team was relieved that the hard part seemed to be over. The next day the employees would use the new system and benefit from the added functionality and the improved throughput. The functionality had been tested thoroughly, so what could possibly go wrong?
The next morning, after the first server crashed due to memory problems and a number of employees complai-ned about the poor interface response times and the very long time it took for every single business transaction, the development team realized that the hard part in fact had just began. Until lunch, three additional servers broke down and the flood of complaints grew. Nobody could
explain the bad performance and stability problems of the new software. The pressure grew from hour to hour, management complained that the employees could not work as expected. Wild guesses were being issued by the stakeholders, each pointing their finger at other depart-ments.
After two days during which no work could be done on the system, management decided to switch back to the old system. Thorough performance analysis of the developed solution showed that the planned architecture was and would never be able to handle the number of requests issued by the users. The whole architecture would need to be revised and re-implemented. Most of the code would have to be adapted.
Another year passed until the software was deployed again - this time thoroughly tested for stability and per-formance. Apart from several small migration issues, the system was running smoothly and the the employees were able to benefit from the new functionality and better performance.
Performance and stability
You can call yourself lucky if you never experienced a scenario like the one just mentioned, but it is well possible that you already have. Application performance and application stability is growing to become the project killer number one of today’s software projects. The same as for functional problems applies to performance and stability issues: The earlier the problem is detected, the cheaper it will be to fix (with respect to time, complexity and money). Unsatisfactory performance often makes it necessary to revise and change the entire architecture - as a result, whole components need to be rewritten and optimized. Thus performance problems usually greatly outpace functional problems in their impact. What complicates matters is that the solution of performance problems usually affects many more components of the software than
com-mon functional defects.
Developing and deploying software without adequate performance management processes can be a considerable risk: maybe it will work, maybe – and more probably – it will not. During the last few years the importance of applica-tion performance management is growing steadily, but many projects are still not implemen-ting an adequate environment to detect performance problems early during development. They are still not creating an adequate load test environment and they are still not setting up tool support to gather information about their application during a load test or during production. With the effect, that if a performance problem arises, first the entire supporting environment needs to be set up, before it is possible
to find and fix the problem. Without a suitable performance setup, the entire performance test environment needs to be built if a problem arises before the problem can be analyzed.
During our consulting assignments we met a number of developers despera-tely in need of tool support to analyze their components from a performance point of view without too much
over-Free application performance management
head. Often requests for performancetools or performance testing is denied by management due to the expected high costs. Managers and developers can now breathe a sigh of relief: NovaTec’s inspectIT will solve their problems for free.
Safeguarding application
performance is not
expen-sive
The stated reason why no performance safeguards are in place is often “high costs“. During project planning, perfor-mance management activities are often forgotten or postponed shortly before deployment on the production system - a few days prior to going live. Often performance tests are not executed at all at this point since the deadline is close. But safeguarding application performance in fact does not cost, but saves money in the long run. Application per-formance management is - like functional testing - a purely risk-driven activity. In fact the sole reason why functional tests are implemented and executed is that the costs (with respect to time, money and quality) are higher if these problems first occur during production by the customer. The budget of functional testing is directly derived from the risk and the impact of functional problems.
The same holds for performance pro-blems. It is naive to think that complex applications will somehow automa-tically provide high performance. By comparing the risks with the impact and taking the reduced risk of perfor-mance problems into account when integrating performance management, a budget for these activities can easily be granted. Performance management is thus not costly, but in fact presents a cheaper and safer approach as it helps reducing the risk of performance
prob-lems and thus the risk of even higher expenses (ranging to and including a premature termination of the project!). There is no “one fits all” performance management solution and not every project needs a high-standard perfor-mance management process. Often performance management is confused with performance testing. Performance management is a process starting with the project initialization, which conti-nues during the complete development lifecycle and supportsthe whole soft-ware lifecycle. Performance testing is one step of performance management during which the actual performance tests are carried out.
It is invaluable that during the definition of the project goals, performance goals are defined as well. The information gained (number of users on the
system, distribution of the load, intents to
increase the sys-tem size after
deployment, detailed
infor-mation of how long each i m p o r t a n t use case may take) will help greatly during defining the software architecture and during the implementation.
Based on this information tests can be set up meaningfully, ensuring that the performance goals can be reached. For these verifications usually tools are necessary.
Available application
performance management
tools
During the last few years suitable cost-efficient alternatives to expensive load generation tools were provided. Pro-jects that do not need the - no doubt
unsurpassed - functionality of the market leaders can still use free, open source tools like “JMeter” or “Grinder” to power load tests.
But generating load on the applica-tion is only the first step. Without the possibility to analyze why and where an application became slow as soon as 50 virtual users perform their pre-defined work, the root cause of the problem cannot be identified and no optimization can be started. Thus load testing without application monitoring does not give any insights into the application and should be considered as a necessary first step for further analysis.
At this point, you know that you will not achieve your goal, but you can still only speculate about the why. We usually see guesswork at this point: “It might be this database statement” or worse, finger pointing between departments or teams: “Our component is running just fine, but component X is always so slow”.
Integrating monitoring within the application will allow the developer to check why the application is getting slower, he is able to directly pinpoint the problem area and knows which part of the application needs to be reviewed and possibly optimized. Until now, cost-effective application monitoring solutions were not available on the market. Thus the project team had to decide whether to use one of the established tools of one of the market leaders or implement own monitoring logic within the code using log output. Unfortunately projects often went the third way and did not implement any performance safeguarding process at all.
The performance testing tools that are currently available can be divided into the following categories:
• Inhouse-implementations and • Professional performance
Why not integrate
perfor-mance measurement tools
into the application code?
The short answer: Flexibility and main-tainability. You will not be as flexible as you will need to be when trying to inspect your application with respect to performance.The more measurement tools you integrate into your software, the less throughput the software will provide. This is due to the fact that collecting the monitoring information also takes time. In fact, including too much monitoring can get you into a situation in which the application performance is so far reduced that the monitoring will not provide accurate performance data. Performance problems can occur any-where within the entire software stack: In-house components, third-party libra-ries, the used application server, data-base queries, the operating system etc. Integrating time measurements only within your application code thus will only cover some of these possible areas.
Isolating performance problems usu-ally includes an iterative refining of the gathered monitoring information. The best solution is to first take a look at the functionality that is proven to be slower than expected and add measurement points to the components called within this functionality. If this information is not sufficient, refine the monitoring and include additional measurement points.
As soon as the performance problem has been isolated and fixed, at least one monitoring point that easily allows checking for this problem again should be integrated. Then this configuration should be stored and all other measu-rement points be deleted.
With manually integrated code to mea-sure your application components, the time to define, refine and remove
monitoring is very time-consuming. Aside from that, the integration of non-business-related code always complicates the readability and thus maintainability. Different open source frameworks provide support to integ-rate monitoring directly with the code and allow basic visualization of the results. The most well-known open source framework is JaMON.
We often see the approach to integrate monitoring and/or performance infor-mation with the standard logging pro-cess (based on tools like the famous log4j). In most cases this approach is chosen to be able to implement “some quick and dirty” m o n i t o r i n g facility to fix the problems at hand. We have seen multiple pro-jects in which a small performance logging approach like this grew to become the common approach for the entire project.
We strongly recommend not to integ-rate performance monitoring directly within the source code. It is inflexible and cannot provide all the information you need to find performance
prob-lems - in addition, usually only time monitoring can be performed.
At the other end of the scale are professional solutions to performance management - the commercial tools, like for example CA Wily Introscope, AppDynamics or dynaTrace.
Nearly all tools provide the perfor-mance analyst with the necessary functionality to inspect and test an application with respect to perfor-mance. But all software solutions in this field also imply high costs. These tools provide unsurpassed functionality, are thoroughly tested and provide support for nearly all environments (even exotic ones). Still many projects just cannot afford to buy an expensive tool to monitor the application and pinpoint performance problems. Due to this NovaTec GmbH proudly announces the release of inspectIT. inspectIT is a Java monitoring tool which focuses on the core functionality necessary to cope with performance problems and provides transparent application monitoring for load test support. Using inspectIT at least 80 % of all perfor-mance management requirements can be easily verified.
Illustration 1 – Not recommended: application monitoring by means of logging
Important aspects of an
application that should be
monitored
We strongly need to differentiate bet-ween monitoring a productive applica-tion and trying to pinpoint performance problems.
For monitoring an already productive application which does not impose severe performance problems, a mini-mal set of monitoring points should be defined, thus allowing for trend analysis. The focus of this method is to detect deviations from the usual behavior and fix them before the users of the application are affected by the problem. “Less is more” is the key. The optimal monitoring set contains as little information as possible to still provide all the necessary information to monitor the application and to find the problem at hand.
For pinpointing existing performance problems or in-depth performance analysis, the amount of measurement points can be increased. For this scenario the tool should help to pro-vide as much information as possible about the internal processes of the application and their performance data - still without causing too much overhead. Gathering all available information is also contra-productive as this will cause too much overhead and falsifes the gathered information too much to be of any help (this is the common problem of profiler tools). There is certain essential information that should always be gathered while testing an application for performance:
Inspect the resources
All applications need resources during runtime. Increasing the load of an application will most likely also increase the load on the resources (or increase the amount of necessary resources). As soon as the request for a specific resource exceeds its availability, the request will be put on
hold - thus decreasing performance of the application. Due to this close inter-dependency between resource and application performance, significant resources need to be monitored. As soon as monitoring shows that resour-ces get scarce, adequate actions need to be taken.
The most evident resources are system resources like the CPU and memory utilization or database resources like a database connection. Resources are also often pooled (e.g. in an applica-tion server), thus these pools need to
be monitored as well.
With inspectIT the monitoring of system resources will be provided directly out of the box. The inspectIT agent directly accesses the platform it is running on and gathers information about the resources of the operating system and about the Java virtual machine.
Inspect the application
With application monitoring you are able to open the blackbox of the Java virtual machine and collect information
Illustration 2 – Monitoring of system resources (CPU and memory) with inspectIT
about your application during runtime. As each application is different the amount and the locations to monitor differ as well. The most common approach is to monitor at least the business-critical interfaces of your components. Depending on your need for information more monitoring points can be inserted to gather more infor-mation about your application.
With inspectIT you can easily place a time-measuring point on any method of any class within your application (or even all third party libraries as long as they are also written in Java). This measuring point provides you with the information about the frequency of and the execution time of the respected method.
Using this approach you can easily monitor the execution count of your components and stay informed about
all execution times of your methods. By placing a measuring point on the entry method of an user transaction you can directly monitor the time your application needs to answer the user request. This information will - like all information within inspectIT - be stored in a database for further analysis. Based on this information trends can be anticipated and actions can be taken.
Inspect the application flow
For in-depth analysis of use cases it is necessary to get a precise understan-ding what the application does and which steps within the application flow are slow. Unfortunately, the application flow is often rather complex and spans a huge tree structure. The flow path through this tree is often closely rela-ted to the input variables. This often
leads to situations in which certain use-cases are infrequently slow or at least differ greatly in their performance depending on the input.
For such situations the best analysis mechanism is to treat each invocation of the use case as a separate set of information. To do so, the tool must provide the unaggregated application flow for each call of the use case. inspectIT implements this concept with its invocation sequence tracer. This functionality allows to define the starting point of a use case and will track each invocation of this method independently.
This gives you the option of performing an analysis with a focus on user trans-actions. The measuring points within an invocation sequence do not repre-sent average values, making them
database is abstracted by additional frameworks. The developers will often never see the actual query executed by the framework. Another frequent scenario is based on the fact that defi-ning new database queries for each perfectly suited for analysis. The fact
that the invocation sequence shows you the specific flow through your application lets you see precisely and in detail at which point your application was slowed down by a specific action. inspectIT sensors know in what invo-cation sequence they were invoked. That’s why inspectIT can offer you extensive navigation options to and from invocation sequences for analy-zing performance problems.
You start for example in the database view that shows all SQL queries that have been run. In this view (average values) you find a database query, which is very slow (high maximum value) at times. Based on this query you can now easily navigate to the invocation sequence (that is, the spe-cific user action) in which this database query was run.
Inspect the database queries
Current estimates put unnecessary or inefficient database queries at one of the top-most positions of performance problems. This usually occurs as the
Illustration 5 – Naviga-tion from the database overview to the invoca-tion sequence view
Illustration 6 – Monitoring of database queries with invocation parameters Illustration 4 – Invocation sequences enable a transaction-based view of requests
An invocation sequence is created for every invocation of „orderCart-ForCustomer”
and can be easily ana-lyzed in the detailed tree view.
persistence framework takes up time. Developers thus tend to re-use existing queries even if these queries return far too much information. But writing new queries results in implementing new code, ranging from new interfaces to new data transfer objects.
inspectIT allows to track all database queries that were created and execu-ted within your application. By default the database tracing is activated out of the box with no manual configura-tion being necessary. The provided information allows to quickly inspect all queries with respect to their execution frequency and their respective execu-tion time. Since all data is stored within a database, trends become visible. In addition inspectIT allows to display SQL information directly within invo-cation sequences easily showing if a method was slow due to complex and inefficient algorithm or due to invoked database queries.
Based on the specific structure of the SQL – which also lets you see concrete bind values for prepared statements – you can now analyze why this query presents a problem. You can further copy this query and send it to your database administrator together with all the required information.
Inspect the architecture
Another significant use of monitoring tools is to evaluate the adherence to architecture specification.
Instrumen-ting the entry methods of the layers and components of the application with an invocation sensor allows to easily check which layers and components were used by each use case.
This information is invaluable for architects to check whether the dependencies between applications are suitable and to understand which components are currently used to implement a use case.
Inspect exceptions
Exceptions that occur in your own program code or in a library in use are an important indicator of the status of your application. It is tedious and error-prone to compile the error codes from a variety of log files in order to analyze
problems with your application. That’s why inspectIT handles the tracking of exceptions for you. The exception sensor is activated the moment errors cases occur in the pro-gram code. In this way errors are made visible directly even if the application falsely ignores them or handles them incorrectly. Identifying these errors provides the basis for improving the application’s stability.
The exception sensor also shows you precisely which components of your application triggered the exception. It thus indicates whether the exception was for instance intercepted at a spe-cific location or whether it was relayed further. Of course inspectIT also provi-des you with the entire stack trace to
Illustration 7 – Monitoring of exceptions
let you analyze the errors in detail. For added convenience you can also navigate easily and quickly from the exceptions overview of the applica-tion to the corresponding invocaapplica-tion sequences.
Inspect HTTP requests
Many of today’s business applications offer access to a web-based user interface via a browser. Knowing which action triggered a specific HTTP request at the back end is of key importance for analyzing a web-based application.
For this reason inspectIT offers a fully configured HTTP sensor that groups HTTP requests by URIs and depicts them in an URI-based aggregated view. This makes it easy to associate specific HTTP requests with the corre-sponding invoca¬tion sequences and to analyze them.
Because the URI of an HTTP request cannot always be associated one-to-one with a specific use case, inspectIT
offers the option of setting a specific use case’s identification flag in the header of the HTTP request. This can for example be configured in a load testing tool such as JMeter to make it possible to associate specific HTTP requests with a use case. inspectIT compiles these HTTP requests in a use-case-based aggregated view. Of course it is also possible to return from both the use-case and the URI view to the corresponding invocation sequences for further analysis.
inspectIT helps you solve
your performance problems
NovaTec incorporates years of experi-ence in dealing with application perfor-mance ranging from the definition and integration of performance processes to safeguard the development of appli-cation to firefights in critical projects to improve performance in a short time. We are proud to offer tool independent performance consulting. To offer the best possible performance consultingwe have developed a strong relation-ship with the two big players in today’s application performance markets: dynaTrace, AppDynamics and CA Wily Introscope. Our engineers provide years of experience in either of these tools and most of the other tools available on the market.
Since 2007 NovaTec engineers of the business unit “Application Per-formance Management” have been implementing a solution to application performance monitoring and thus we are proud to officially announce the first official release of this application, inspectIT. The development of inspec-tIT includes performance monitoring experience gathered from many highly skilled NovaTec engineers to help you solve your performance problems. inspectIT works by transparently integrating callback hooks into the application using load-time Java byte code modification. These hooks are then used by inspectIT’s sensors to collect various monitoring information about the application. This monitoring approach is integrated with a minimum
of (transparent) code adaptation and thus is the safest possible way to integrate monitoring. All measurement logic is extracted to the sensors that run “outside” the business relevant logic code and in this way is safely separated from your application. The flexible and solid core framework of inspectIT allows easy integration of additional sensors. Hence new sen-sors can be created to collect the infor-mation you need for your application. An inspectIT installation consists
of at least one agent running in the application JVM and monitoring the application(s), one (or quite rarely multiple) central measurement reposi-tories (CMR) that store the data sent by the agents to a persistent storage and multiple clients that are connected to the CMR and display gathered and aggregated data in a significant way. The inspectIT agent is the component that enables measuring points within the monitored application. It must be integrated into the startup script of your application (or application server) and
automatically instruments the configu-red methods with the defined sensors. All configuration settings are defined within a configuration file residing in the agent configuration folder. The integration of the agent is as simple as adding one additional JVM option to the startup script of your application. Setting up the CMR and the GUI is even simpler - just copy the releases to the preferred location and execute the startup scripts.
Illustration 9 – inspectIT components for application monitoring
User Interface
User Interface
CMR
Server
App 1
App 2
inspect
IT
Agent
App 3
Interested?
Contact us at [email protected] or call us at +49 (711) 22040 700. You may also use the contact information provided below.
M.Sc. Stefan Siegl
Business Unit Manager “Application Performance Management” Project lead “inspectIT”
M.Sc. Patrice Bouillet
Lead architect of “inspectIT” [email protected]
Factsheet
inspect
IT
Free performance diagnostics
Performance analysis doesn‘t have to be expensive. inspectIT gives you everything you need to carry out performance diagnostics and monitoring on your applications ? and what‘s more, it‘s free.
Your Java application need no longer be a mysterious black box. Use inspectIT to unlock what‘s under the lid and give you the upper hand in the struggle for application performance, right from the development phase. With inspectIT you can see what‘s going on inside your application, allowing you to identify and fix performance issues. Win the battle for performance by integrating transparent inspectIT sensors, with no need to modify your program code.
Integrating measurement points is as simple as adding a single line in a configuration file. Save time, effort and money by identifying potential performance bottlenecks before delivering the application to the client. When it comes to identifying and fixing performance issues, sooner is definitely better. Enjoy improved application performance thanks to inspectIT
Features
• Fast and simple installation and configuration
Start implementing your applications without complex and time-consuming tool setup. inspectIT can be set up in a matter of an hour. Afterwards you can start defining your measurement points and start the inspection.
•
Seamless integration
inspectIT integrates itself seamlessly in any Java application by means of byte code instrumentation. During classloa-ding the measuring code is added, so your business code stays exactly the same, no business logic is altered in any way.
• State-of-the-art technologies and use of official standards
inspectIT is written completely in Java and only uses officially released and standardized interfaces. All components of inspectIT are based on well-known and standardized libraries like Spring, Eclipse RCP, PicoAgent, ...
•
Decentralized architecture
The flexible decentralized architecture allows having one (or multiple) central servers collecting, storing and aggregating the data of multiple agents and providing this data to multiple standalone clients for analysis.
•
Optimized for low overhead
inspectIT is designed to run with as little overhead as possible even in complex clustered environments.
•
Monitoring Functionality
inspectIT can be used to monitor your live application and provide invaluable information about the current status of your application.
•
Flexible architecture allows fast integration of sensors
inspectIT is based on an highly flexible agent architecture that allows seamless integration of sensors into the appli-cation code. A vast number of supported sensors for instant usage is available out-of-the-box. Additional sensors can easily be integrated. Currently inspectIT provides the following sensors:
Additional sensors can be integrated easily. Currently inspectIT contains the following sensors:
Time measurement (Timer sensor)
Easily monitor the execution times of methods within your application to see which methods use up the most time. The sensor is also able to provide the pure CPU times which allows checking whether the method took long due to complex logic or if it was merely on hold.
Platform Sensor
The platform sensor allows monitoring of system resources like CPU utilization, memory utilization, threading etc.
Database Sensor
Performance problems are often related to inefficient or unnecessary database queries. inspectIT allows to monitor the duration and structure of all database queries and provides the query as it is sent to the database even with all parameters flled in.
Invocation Sequence Sensor
Use invocation sequences to get unaggregated call invocation trees. Using this sensor you will directly get the information which methods were called by the users and which methods took long within this execution of the logic. This functionality is invaluable for inspecting invocations that are slow sporadically or in-depth analysis of certain functionality.
Exception Sensor
The exception sensor is activated the moment error cases occur in the program code. In this way errors are made visible directly even if the application falsely ignores them or handles them incorrectly. Identifying these errors provides the basis for improving the application’s stability.
HTTP Sensor
The HTTP sensor makes it easy to associate HTTP requests and use cases with invocation sequences. This allows a powerful form of analysis of web applications.
• Flexible possibilities for sensor configuration
inspectIT incorporates a flexible configuration of sensors.
•
Architecture Validation
Architects can focus the instrumentation to gather architecture validation information, allowing them to verify if the developers use the architecture as the architect designed it to be used.
•
Visualization
A highly configurable state-of-the-art user interface allows easy analysis of the gathered data.
•
Support for all Java applications on all platforms
inspectIT can monitor all Java applications (JVM version > 1.3) and works with all known application servers.
•
The free Java monitoring solution
inspectIT provides all functionality necessary to implement application monitoring for Java applications but, unlike other other tools, is free.
www.inspectIT.eu
[email protected]
© 2009-2013 NovaTec - Consulting GmbH
All other brands or product names mentioned here are trademarks
or registered trademarks of the respective owners. Version 1.3 / März 2013
