Part 3 - Performance: How to Fine-tune Your ODM Solution. An InformationWeek Webcast Sponsored by

(1)

An InformationWeek Webcast

Part 3 - Performance:

(2)

(3)

Today’s Presenters

David Granshaw

WODM Performance Architect (Events)

Pierre-André Paumelle

(4)

Demo Series: Best Practices for Implementing

Operational Decision Management Solutions

Part 3: Performance: How to Fine-tune Your ODM

Solution

(5)

WebSphere Decision Center

WebSphere Decision Server

WebSphere Operational Decision Management

Event Execution Runtime Event Designer

Decision Center Console

Decision Center Repository

Decision Center for Business Space

Rule Execution Server

Rule Designer

Relations between components

Rule Solutions for Office 5 Ruledocs Deploy Deploy Deploy Deploy Synchronize Synchronize Session 2 focus Session 1 focus Session 3 focus

(6)

► WebSphere Operational Decision Management

► 1. Tuning Decision Server Events

► 2. Tuning Decision Server Rules

► 3. Tuning Decision Center

► 4. Performance Improvements in V7.5

Agenda

Note the performance figures given in this presentation are just examples based on a

typical workload. The performance impact of specific design and runtime changes will vary depending upon the details of the workload.

(7)

Decision Server Events – Runtime Overview

Actions Events Repository Database DB Server JMS (durable) Event Destination JMS (durable) Action Topic Events Events Runtime WODM Server Technology Connectors Technology Connectors

All events and actions ultimately end up on the JMS event and action destinations (except events delivered via Extreme Scale).

Technology Connectors can be used to send actions and receive events from a range of sources including JMS, File, HTTP, JDBC etc

The Events Repository can be used to store information about the occurrence of events, history and any data that may be shared across events (CSBOs).

(8)

Event Designer : Durability of Events and Actions

Non Durable processing can be 2.5 times faster than durable

processing so if actions and events do not need to survive a

server restart use non durable processing

For further information on durability see the WODM Infocenter : “Configuring

durable and nondurable events and actions

”

(9)

Event Designer : Simple and Complex Event Rules

• Simple event rules

– By definition, Simple event rules can be evaluated without reference to previous events or actions and are therefore significantly faster

• e.g. “if the deposit amount is > $500 then …”

• Complex Event Rules

– By definition, Complex event rules have a dependency on previous events or actions with the same context ID (e.g. customer reference number)

• e.g. “If this customer has withdrawn money 3 times in the last week then …”

• For optimal performance do not add a Context Id to a Simple event rule

unless required.

Simple rules should only have a context ID defined if:

• Information about the occurrence of the events and action processed by this event rule are actually required by other, Complex event rules.

(10)

Event Designer : Simple Rules versus Complex Rules

1.87 1.41 1 0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 2 R e la ti v e Th roug hp ut ( e v ts /s ) Non Durable Simple Rule

Simple Rule with Ctx ID Complex Rule

Simple event rules are faster than Complex rules by:

Non Durable Processing: 87% Durable Processing 22%

1.22 1.11 1 0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 2 R e la ti v e Th roug hp ut ( e v ts /s ) Durable Simple Rule

Simple Rule with Ctx ID Complex Rule

Don’t define a context ID on a Simple rule unless required

(11)

Event Designer : Ordering Multiple Expressions/Filters in an Event Rule

• Rules will evaluate expressions intelligently from the top down, so

place Simple expressions first

– For example, the second part of an ‘and’ expression will only be evaluated if the first part is true.

Complex expression first :

More costly Complex expression is always evaluated

Simple expression first :

More costly Complex expression is only evaluated when the Simple expression is true

(12)

Event Designer : Ordering multiple filters in a rule

1.51 1 0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 R e la ti v e Th roug hp ut ( e v ts /s ) Non Durable

Simple Filter First Complex Filter First

1.11 1 0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 R e la ti v e Th roug hp ut ( e v ts /s ) Durable

Simple Filter First Complex Filter First

Placing Simple filters first may be faster by:

Non Durable Processing: 51% Durable Processing 11%

Note: The performance difference will depend how often the first filter evaluates to true. In the graphs above the first filter evaluated to true 1 in every 100 events.

(13)

Event Designer : Ordering multiple filters in a rule

• For optimal performance:

Always place Simple filters ahead of Complex filters.

– The same principle applies to any filters that require more processing, for example, simplistic filters should come before filters that involve database fetches or Context Scoped Business Objects

Also, if an event triggers multiple Complex rules to be evaluated, group the rules together into one event rule group.

(14)

Event Designer : Technology Connectors

These examples show an example of the performance impact of using the new JMS and HTTP connectors shipped in V7.5.

1 0.7 0.7 0 0.2 0.4 0.6 0.8 1 1.2 R e la ti v e Th roug hp ut ( e v ts /s ) Event Connector No Connector HTTP Connector JMS Connector 1 0.8 0.9 0 0.2 0.4 0.6 0.8 1 1.2 R e la ti v e Th roug hp ut ( e v ts /s ) Action Connector No Connector HTTP Connector JMS Connector

Event Connector Action Connector

For optimal performance, send events in the correct format directly to Decision Server JMS event or durable event destinations. Also receive actions directly from the action or durable action topic

(15)

Event Runtime Performance : Top Ten Recommendations

1. Monitor system resources (cpu, disk, network, memory)

• This will ensure you target your performance tuning effectively

2. Check you are not writing large amounts of data to log, error log, ffdc or

trace files

• It will significantly impact performance and may indicate an underlying problem in the project or configuration, which will degrade throughput.

3. Ensure you are not recording history unless you intend to.

• History is needed for user defined charts or reports of events, actions, filters and data. It is not needed for Complex rule evaluation or Context Scoped Business Objects.

• History records large amounts of data. It can impact performance by 50 to 80%.

(16)

4. Tune the JVM

• A heap size of 1280MB is a good starting point for 32 bit systems; 4096MB for 64bit.

• Consider using generational garbage collection

5. Ensure you have a sufficient number of rule processing threads

• You should consider increasing the number of rule processing threads if events are building up on the event destination

• If you increase the number of rule processing threads you also need to consider increasing the connection pools for database connections, JMS Connections and data connection objects

6. Periodically (either manually or automatically) prune unnecessary

context data from the Decision Server Event Repository to improve the

database performance

(17)

7. Ensure the repository database is tuned (or auto tuned) for the workload

– use separate, fast disk subsystems (SAN/RAID) for all logs and tables.

8. When using persistent Context Scoped Business Objects, the following

database tuning may improve performance up to a factor of 3:

– For db2 consider using the statement concentrator:

• UPDATE DB CFG USING STMT_CONC LITERALS IMMEDIATE; – For Oracle consider using cursor sharing:

• ALTER SYSTEM SET CURSOR_SHARING=FORCE SCOPE=BOTH;

9. Tune the JMS provider and monitor the queue/topic message depths

10. Use the Performance Monitoring Infrastructure to monitor Decision

Server Events

• This will show statistics about events, actions, rules and other aspects of the runtime. See WODM Infocenter for more details

(18)

Note the next 4 slides were not presented at the webinar but provide

additional detail on some of the top ten tuning recommendations.

(19)

Runtime : Minimising Disk Access - logging

WODM has a number of log and error files. If these are growing significantly over time it can have a large impact on performance and may indicate an underlying problem.

Check that the following files and ensure they are not growing

significantly over time. If they are, read the logs to find the cause and rectify the problem:

• <was_install_dir>\profiles\ <profileName> \logs\SystemOut.log • <was_install_dir>\profiles\ <profileName> \logs\SystemErr.log • <was_install_dir>\profiles\ <profileName> \ffdc\ (all files)

Also confirm that large trace files are not being generated in

• <was_install_dir>\profiles\<profileName>\logs\trace.out

• <DecisionServer_install_dir>\director\logs\connectors.log (standalone connectors only)

For further information on trace see the WODM Infocenter : “enabling trace” and “Configuring the technology connectors log file”.

(20)

Runtime : JVM Tuning

• The tuning of the JVM and the garbage collection policy can have a

significant impact on performance.

• Settings will depend upon the WODM project and the OS but here are

some recommended starting configurations

For further information on JVM tuning see the WAS Infocenter : “JVM Tuning” 32 bit

WODM installation

64 bit

WODM installation Minimum heap size 1280 (MB) 4096 (MB)

Maximum heap size 1280 (MB) 4096 (MB) Generic JVM arguments -Xgcpolicy:gencon –Xmn1024M -Xgcpolicy:gencon –Xmn2048M

(21)

Runtime : Decision Server Events Tuning

• If there is spare cpu on the WODM server and events are building up

on the event destination this may indicate an insufficient number of

rule processing threads. The number of threads can be increased in

the WAS administrative console:

– Resources > Resource Environment > Resource environment entries > WbeSrv01 > Custom properties

– In a cluster it is recommended to use a prime number for the number of threads. This will provide the greatest concurrency.

(22)

Runtime : Decision Server Events Tuning

• If you do increase the number of rule processor threads:

– As each rule processor may use a database connection, ensure there are a sufficient number of JDBC threads

• Resources > JDBC > Data sources > Event Runtime Datasource > Connection pools

– Set “Maximum connections” to “rule processor threads + history MDB concurrency + 10”.

– As each rule processor thread may use a JMS connection, ensure there are a sufficient number of JMS connections

• Resources > JMS > Topic connection factories > WbeTopicConnectionFactory > Connection pools

– Set “Maximum connections” to “rule processor threads + 10”

– When using database enrichment ensure there is a connection available for each rule processor thread. This is set in event designer data

connections page.

(23)

HA, Scalability and Performance

• When using WebSphere Default Messaging for JMS

– There are two key clustering architectures mentioned in the WODM Infocenter, the “Silver Topology” and the “Golden Topology”.

– The Golden topology has separate messaging and WODM clusters

– The Silver topology contains a messaging engine and a Decision Server Runtime in each cluster member

– There is also a choice of messaging engine configurations

– The scalability options will offer the opportunity to have multiple messaging engines and will provide the greatest throughput.

For optimal performance use the Silver Topology with a “Scalability” or

“High availability with Scalability” messaging engine configuration.

(24)

HA, Scalability and Performance – Golden versus Silver

Topology

1.8 1 0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 R e la ti v e Th roug hp ut ( e v ts /s ) Non Durable Silver Topology Golden Topology 1.4 1 0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 R e la ti v e Th roug hp ut ( e v ts /s ) Durable Silver Topology Golden Topology

The Silver topology can out perform the Golden topology by 40 – 80% due

to the co-location of the messaging engine and events runtimes

(25)

► WebSphere Operational Decision Management

► 1. Tuning Decision Server Events

► 2. Tuning Decision Server Rules

► 3. Tuning Decision Center

► 4. Performance Improvements in V7.5

(26)

26

Decision Service architecture

Rule Engine Execution

IN IN OUT Ru le flow BOM to XOM XOM Rules

• BOM: Business Object Model • XOM: eXecutable Object Model

(27)

27

Decision Service performance best practices

• The performance cost for a Decision Service may look

something like

Execution Time _XOM Ruleflow Functions B2X

Misc. Rule engine

JVM (GC)

(28)

Decision Service performance best practices

• XOM type choices:

– JAVA XOM better performance

– XML XOM

• Dynamicity

• Useful in case of XML model

• Ruleflow:

– Limit the size and the complexity of the ruleflow, it is

interpreted.

(29)

Decision Service performance best practices

• Choose the correct engine algorithm depending on

your Decision Service.

– RetePlus (The default mode)

• Stateful application

• Rule chaining application

• May be useful in the case of many objects

– Sequential

• Application with many rules and few objects • Most of the customer cases.

• Really efficient in multi-thread environment. – Fastpath

• Application with rules implementing a decision structure and many objects

• May have longer compilation but faster at run time.

(30)

30

Decision Service performance best practices

– Smallest memory consumption • Engine alone

– No concurrent execution • Engine alone

• JSE Rule Execution Server

– Concurrent executions • Rule Execution Server

JSE

• Rule Execution Server JEE

– Integration with JEE animals (EJB, MDB, POJO)

• Rule Execution Server JEE

– Call as a Web Service • HTDS

• MTDS

– Call from COBOL application • zRES



Choose your architecture depending on your Decision Service

(31)

Decision Server Rules tuning

• The log level in the Rule Execution Server should be set to level

Severe or Warning in the production environment to increase

performance.

– This property (TraceLevel) is accessible in the resource

adaptor of the Rule Execution Server or in the ra.xml.

• Tune the GC and memory size.

– Starting configuration 64bits

– -Xgcpolicy:gencon –Xmn2048M –Xmx4096M –Xms4096M

• Tune the RES pool size. A sizing methodology is available at :

(32)

Decision Server Rules tuning

• Tune the usage of the execution trace and the

Decision Warehouse by filtering.

• A ruleset on an XML XOM should be configured to

run in multiple simultaneous.

– This is configured using the ruleset property

ruleset.xmlDocumentDriverPool.maxSize.

• Use the ruleset caching information (

XU dump

) from

the Rule Execution Server console to get the Rule

Execution Server status.

(33)

Decision Server Rules Performance

T

PS

(34)

Decision Server Rules Performance

T

PS

(35)

Decision Server Rules Performance

T

PS

(36)

Decision Server Rules Performance

T

PS

(37)

► WebSphere Operational Decision Management

► 1. Tuning Decision Server Events

► 2. Tuning Decision Server Rules

►

3. Tuning Decision Center

► 4. Performance Improvements in V7.5

(38)

Decision Center tuning

• Limit memory consumption

– Parsing a ruleset consumes memory so you can disable the

“Archive parsing flag” option (Project > Edit project options >

Check the ruleset archive).

– Use Automatic build to avoid a huge ruleset generation cost

at “first ruleset generation”.

– Set in the installation manager the configuration parameter

teamserver.build.archive.storage to file instead of memory.

• Build performance

– Disable “Archive parsing flag”.

– Disable “Rule analysis checks”.

(39)

Decision Center tuning

• Memory consumption estimation on a repository of

10000

business rules

– Memory foot print of a session is ~10 MB

– Ruleset generation : ~150 MB of short-lived objects created

• If you believe the number of users connected

simultaneously to a single server will not fit into the

memory allocated for a single VM, you should deploy

Decision Center to a cluster of servers and use

load-balancing, so that HTTP sessions are dispatched

according to the server’s available memory.

(40)

► WebSphere Operational Decision Management

► 1. Tuning Decision Server Events

► 2. Tuning Decision Server Rules

► 3. Tuning Decision Center

► 4. Performance Improvements in V7.5

(41)

41

Decision Server Rule

Benchmark Scenarios (relative to 7.1.1.1 GA)

• Rule Execution Server execution improvement (TPS)

Between 1.01x to 3.97x faster than 7.1.1.1

Average improvement factor = 25%

• The new Web Service stack (HTDS) for rulesets based

on java XOM offers performance improvements

(42)

42

Decision Center

(relative to 7.1.1.1 GA)

• Decision Center delivers a new ruleset build chain.

Performance gains from 50% to 150% versus

Rule Team Server V7.1.1.1.

This optimization offers a better scalability of

Decision Center as performance gains increase with

rule project size.

(43)

Customer Based Scenario 1 : Credit Limit Check

Warn Customer Action Withdrawal Events Enrich Event: Get current balance and credit limit from

DB Complex rule: Has balance been exceeded before? Simple rule: Check if limit exceeded

Performance in V7.5 improved by (relative to V7.0.1.1):

18% for non durable processing 10% for durable processing

• This scenario processes withdrawals from a bank and looks for

customers who repeatedly exceed their credit limits. The majority of

processing is with Simple event rules.

Synthetic event Generated 1 in every 100 events

(44)

Customer Based Scenario 2 : Missed Payments

Credit card, overdraft, loan

missed payment

events

Performance in V7.5 improved by:

31% for non durable processing 14% for durable processing

• This scenario looks for customers who have repeatedly missed payments

on multiple accounts : credit card, overdraft and loan. It also uses CSBOs to

keep track of the total of all missed payments. All event rules are Complex.

Warn Customer Action Enrich Event: Get current customer details from DB Complex rule: Has this occurred more than once and

is the total debt > $5000

Complex rule: Have payments been missed on all

accounts

Synthetic event Generated 1 in every 100 events

(45)

Customer Based Scenario 3 : Rules and Events

Offer Customer Discount Action Quote

Requested

Event Complex rule:

Has this customer applied before Simple rule: determine discount from Decision Server Rules

• Performance in V7.5 improved by:

• 27% for non durable processing

• 16% for durable processing

• Scenario 3 is a customer based scenario, which looks for

customers who

are eligible for a discount as part of a new marketing campaign. The

scenario involves DSE and DSR.

Decision Server Rules (DSR) Decision Server Events (DSE)

SOAP Technology Connector

(46)

Where to go for further information

• WebSphere Operational Decision Management – www.ibm.com/operational-decision-management

• Redbook on BRMS Tuning

– Proven Practices for Enhancing Performance: A Q & A for IBM WebSphere ILOG BRMS 7.1

• Whitepaper on WebSphere Business Events Tuning

– Getting the best performance from your complex event processing system with WebSphere Business Events

• DeveloperWorks tool (System Integration Bus Performance) that

automatically engages and displays PMI statistics relevant to Decision Server Event runtime performance

(47)

Q&A Session

David Granshaw

WODM Performance Architect (Events)

Pierre-André Paumelle

(48)

Resources

To view this or other events on-demand please visit:

http://www.informationweek.com/events/past

For more information please visit:

WebSphere Operational Decision Management