Developing Interactive Voice Response
applications on JBoss Enterprise
Middleware: A Case Study
Ravi Srinivasan, Hector Hugo Gonzalez
Hewlett Packard Consulting & Integration Group Feb 14 2008
About us
HP Open Source Middleware - www.hp.com/go/osms Consulting Services
Linux/Apache/Tomcat/JBoss Migration Services
Performance Tuning & Optimization
Outline
Introduction & Overview Solution Architecture Deployment Strategy Performance
Summary
Customer Profile
One of the largest providers of mobile services in
Mexico
More than 45 million subscribers…and growing! 70 million+ subscribers all over Latin America Majority market share in multiple countries in Latin
America
Interactive Voice Response Systems (IVR)
Handle large call volumes efficiently
Efficient segmentation/routing based on caller profile Extended business hours
Cost-effective
The “old” system…
Not scalable
Software approaching EOL Proprietary - integrating with
other systems difficult
Failing to meet SLA – losing revenue!
A solution was needed – and fast!
High performance and meeting SLA’s High Availability/fail-over capabilities Open and scalable platform
Integration capabilities Standards based
Proposed Solution
Replace existing prepaid IVR functionality with VoiceXML
applications developed on HP’s OpenCall Media Platform (OCMP) and JBoss Enterprise Middleware
OCMP provides a new IVR system built on industry
standards that can act as a foundation for future Voice & Media services
JBoss Enterprise Middleware
− Lower TCO. Zero license fees − Open and Scalable platform
VoiceXML
W3C standard for interactive dialog between human
and computer
Interpreted by a “Voice Browser”
*Source: Wikipedia
Solution Architecture
Solution Architecture
Telephony Network HP OpenCall Media Platform as VoiceXML gatewayLegacy / Back office / Prepaid Systems Application Server Database VoiceXML HTTP http://appsvr/ivr/hello.vxml 333 URL Called number TCP/IP TCP/IP
Application Server VoiceXML Browser http request VoiceXML document TCP/IP Sockets XML-RPC TCP/IP sockets CSV JDBC
Java Virtual Machine
JBoss Application Server Static VXML documents
Apache Web Server IVR Application
OCMP
Controller Servlets
controller view model XML XSL JSP DAO DAO DAO HTML Browser http request HTML
Design Talking Points
To EJB or not to EJB… Multiple legacy back-ends
− CSV formatted results over TCP/IP sockets
Multiple third party external systems
− XML/RPC
− JDBC
Coarse grained Stateless services
− Easier to provide HA facilities
− Performance achieved by scale out
Document Types
Static Documents (Apache)
VXML documents that will not change, for instance:
menus, application logic, etc.
Audio files.
Grammar files (used for user input validation). Dynamic Documents (JBoss)
VXML documents built on-the-fly (XSLT), for instance:
Balance Enquiries and Credit Replenishment results.
Data fetched from back-end (RDBMS, Legacy etc) – low
level re-usable libraries to each of the back-ends
Central idea – Redundancy
Redundancy across all layers OCMP
App Server Backend
Hardware Redundancy RAID
NIC bonding (“teaming”)
Redundant PSU’s/Routers/Switches
Approach 1: HA / Failover using JBoss Clustering
Freely available out of the box. Very flexible (dynamic cluster
membership)
Stability and performance issues with mod_jk
Moving to Apache mod_proxy/mod_rewrite. Better stability
and performance
Approach 2: HA / Failover using HP ServiceGuard
HP ServiceGuard is an enterprise grade HA solution that is
widely used for mission critical IT infrastructures
Migrates IP of failed node to a Standby Active/Passive node Flexible cluster configuration (Active-Active, Active-Passive
HA using HP ServiceGuard
Disk Array (Raid 1) HP Service Guard HP Service Guard HP Service Guard HP Service Guard CLUSTER CLUSTER CLUSTER CLUSTER JBoss1 (Active) AppSvr1 AppSvr1 AppSvr1 AppSvr1 Red Hat Enterprise Linux AS 4
JBoss2 (Active/Passive) Intra-cluster communication AppSvr2 AppSvr2 AppSvr2 AppSvr2 Red Hat Enterprise Linux AS 4
Management & Monitoring
HP OpenView for monitoring (high-level) Plugins for JBoss, Tomcat, Apache & MySQL
available for free!
http://www.openview.hp.com/products/spi/prod_spi_0002.html
Custom MBeans exposes some run-time data
HP SIM (System Insight Manager) for Rack/Blade level
monitoring
Customized Redhat OS
Customized “software appliance” like stack Slimmed down version of Redhat Linux AS 4 Quick & Easy Provisioning
Image based (for homogeneous hardware) Kickstart
LinuxCOEhttp://linuxcoe.sourceforge.net/
Patches and Updates on a “need only” basis
Customized JBoss
Customized for fast startup (15-20 seconds) Custom Logging policies
Minimalist jboss-service.xml
Synchronized deploy directories via NFS/FTP Patches and Updates on a “need only” basis
Performance
Performance focus areas
Apache/JBoss OS & Hardware JVM
Application Source Code
Apache/JBoss
Tune for higher concurrency MaxClients
ServerLimit
maxThreads (JBoss/Tomcat) Disable unwanted modules/services
Access logging – separate partition (mount “noatime”) Offload SSL processing to dedicated hardware
Prefer mod_proxy_* over mod_jk for Apache 2.1/2.2 Apache KeepAlive timeout – very low or turn OFF
Note on JBoss Logging
Log Levels – prefer ERROR or WARN (in Production) Configure log file to “rollover” after it reaches a
pre-set size
Avoid using NFS for log directories
Separate partition – mount with “noatime” option Log4j jars inside your WAR – can cause problems
use “scoped” logging config use JDK java.util.logging
OS & Hardware
Hardware
Fast CPU’s with large L2 caches 15K RPM SCSI disks, plentiful RAM Gigabit network switches
File descriptors – fs.file-max, ulimit –n (per process) TCP stack tuning
rmem_default, rmem_max, wmem_max,
wmem_default
tcp_sack, tcp_timestamps OFF (YMMV)
JVM
Aim for stability first!
Server side apps – tune for throughput (mostly) Important params (YMMV)
-Xmx, -Xms, Eden space/ Survivor space sizing UseParNewGC (for Eden), CMS collector (for
tenured)
PermSize (to avoid OOM errors)
JVM Tools
VisualGC/jconsole (Sun JVM)
JRockit Mission Control (BEA JRockit) HPJTune (Free!)
http://www.hp.com/products1/unix/java/java2/hpjtune/
Detailed Code/JVM Profiling Yourkit
JXInsight
Last but not least….application source code
Profiling revealed bottlenecks in the DAO->backend
layer
Socket creation and teardown very expensive Thread dumps (SIGKILL) showed many threads
waiting - contention on shared resources
Socket pooling improved performance tremendously Processing Read only Transactions from a separate
dedicated back-end node also improved performance tremendously
Performance numbers
approx. 24 million requests served per day Stability of the middleware platform has been
excellent
< 30 seconds fail-over and recovery (Active-Passive) 30 million subscribers migrated over with minimal
disruption
Summary
In conclusion…
Stability and Performance of the system has resulted
in customer’s SLA being met
Mission critical IVR applications can be scaled on
commodity hardware and open platforms like the JBoss Enterprise platform
Reduced TCO for the customer by using open source
software and commodity hardware
The open and standards compliant nature of OCMP
and JBoss makes it easy to add new functionality and services
Next Steps…
We are already in the process of implementing new
services on the JBoss Enterprise platform
We expect technologies like Seam, JBoss Messaging
and ESB to play a very important role in the future
Evaluating jBPM and Drools for workflow services We are planning to implement JAX-WS style web
