IA6.5 Performance Tuning

(1)

TAKING ADVANTAGE OF

THE EMC CAPTIVA

ARCHITECTURE

Applying Best Practices to Optimize

Performance

Christopher Lund EMC

(2)

Agenda

• How InputAccel works

• InputAccel 6.5 Benchmark Results • Tuning – InputAccel Server – Batches – InputAccel Database – InputAccel Database – Client Modules – Capture Workflow

• Diagnosing Performance Issues

The EMC® _Captiva® _InputAccel® _{and Dispatcher}™_{Version 6.5 Performance Sizing}

and Tuning Guide – which is available on PowerLink – provided much of the data for this presentation.

(3)

A multi-machine capture application server

• Server is the data tier (memory

mapped)

• Server manages task queues • Server is multi-threaded…

• VBA execution is single-threaded

How InputAccel Works

InputAccel System Export Modules Export Modules Processing Modules Processing Modules Capture Modules Capture Modules Capture Modules Export Modules Processing Modules •

• DB writes are queued, but

single-threaded

• Server uses asynchronous I/O • Most work done from a thread

pool

• Clients are the executing tier

(where scaling comes from)

InputAccel Servers

WIP &

(4)

How InputAccel Works

An execution pipeline

• Task queues are not FIFO

• Tasks are scheduled based on Priority, then creation date

• Recovery is through reprocessing Execution Pipeline B2 B3 B4 B3 B4 B4 t1 t2 t3 m1 m2 m3 m4 reprocessing • Not a repository

• Assume short duration tasks • Work is pushed, no polling • Tasks may be prefetched

B1 B1 B2 B2 B3 B3 B4 A4 B1 B2 A4 B1 A3 A4 A2 A3 A4 A2 A1 B2 B3 B4 A3 A2 A1 t3 t5 t4 t7 t6 t8 t9

(5)

How InputAccel Works

Like a Petri net

• A ProcessFlow defines the steps

and trigger levels.

• Implicit fire when data is available • There is no predefined execution

Petri Net

A

• There is no predefined execution

order

• There is no end state – IADone

triggered implies completed

D

C

B C

(6)

InputAccel 6.5 Benchmark Results

• One 8-core IA Server, over 300 Client Modules • In-house, ideal conditions, your mileage will vary • Performance similar to IA 6.0 SP1

Performance Level Overall Task Processing Rate (tasks/hour) Processing Rate/CPU Core (tasks/hour) Avg. CPU Utilization/C PU Core

(tasks/hour) (tasks/hour) PU Core 50 active batches w/reporting disabled 2,672,007 324,001 67% 1000 active batches w/reporting disabled 1,990,892 248,862 53% 1000 active batches w/reporting enabled 1,384,910 173,114 32%

(7)

Tuning – InputAccel Server…

• It is a server application. It will use all available resources

• Recommendation: use Windows 2008 R2 for best performance • CPU

– ias.exe is multi-threaded

– Recommendation:use at least a 4-core CPU for optimum throughput, 8+ is

better

• RAM • RAM

– Recommendation:4-8 GB RAM, no less than 4 GB – ias.exe is a 32-bit app, so 4 GB address space max

– BatchMaxAddressSpaceK controls how much RAM IAS uses

• 1.5 GB is the default

• Set to 2 to 2.5 GB on 32-bit Windows with /3G option • Set to 3 to 3.5 GB on 64-bit Windows

– Only as many batches as will fit in BatchMaxAddressSpaceK are kept in RAM,

when there are more, swapping occurs

• Have your working set of batches small enough to all fit in RAM

(8)

…Tuning – InputAccel Server

• Disk

– Used heavily for batches and processes

– RAID 1+0 is best – RAID 5 usually is not fast enough and is not recommended

• Recommendation:use a caching controller with Read Ahead and Write Back

– SAN is OK, NAS not recommended

– Turn off anti-virus scanning on IAS folder

– IAS folder should be on a dedicated disk drive or array so it is not shared by other

programs or the Windows swap file programs or the Windows swap file

– #2 cause of slow performance: a slow hard drive • Network

– InputAccel has a “chatty” protocol between client modules and the InputAccel Server

– For best performance, client modules, the InputAccel Server, Administration Console, and

DB should be on the same sub-net

– WANs usually have low bandwidth with high latency, which may make it unsuitable

• Connecting client modules to InputAccel Server via a WAN is doable with adequate performance – depends on the client modules, # batches on IA server, and IPP

(9)

Tuning – Batches…

• Batch size (IAB file)

– 5 – 20 MB is ideal – < 100 MB is OK

– > 100 MB not recommended, but is allowed – 10 – 100 pages per batch recommended

– 1000 pages per batch degrades throughput by 10% or

more

– < 10 pages per batch leads to too many small batches and

(10)

…Tuning – Batches

• Number of Batches

– Throughput is best when the working set of batches fit in RAM • Typically 50 – 500 fit depending on batch size and

BatchMaxAddressSpaceK

– When there are more batches than fit in RAM, IA Server swaps batches in/out as needed

• Swapping decreases IA Server throughput

– Up to 9,000 idle batches is possible with adequate performance – Up to 9,000 idle batches is possible with adequate performance

• Idle means that all tasks within the batch have been processed and

finished

– IA Server startup is slower with 1000’s of batches because IAS must load all batches into RAM to extract data

• After startup they are swapped to disk and do not consume too many

resources

– #1 cause of slow performance: too many active batches causing excessive swapping

(11)

Tuning – InputAccel Database

System Requirements

• InputAccel Server requires a database to run

• Only MSSQL is supported, versions 2005 and 2008

– Express supported only for low volume and no IA Reporting

– Standard or Enterprise recommended for medium to high volume

• IADB stores:

– IA configuration data

– Reporting data on completed batches/tasks – Work In Progress (WIP) status

(12)

Tuning – InputAccel Database

System Requirements

• Recommendation: 64-bit MSSQL server for best performance

– 2-4 CPU cores – 4-8 GB RAM

– RAID 1+0 with read/write caching controller w/ fast disks (15k RPM)

• InputAccel Server requires fast, uninterrupted access to the DB • InputAccel Server requires fast, uninterrupted access to the DB

– If the DB goes offline, IA Server pauses

• Recommendation: put the DB and InputAccel Server on the

same low-latency, high-bandwidth subnet

• Reporting decreases the InputAccel Server throughput by about

(13)

Tuning – InputAccel Database

Data Volume and Rates

• IADB File Size and Growth Rates

– Configuration data - typically < 100 MB

– WIP data - 1 MB × # batches

• WIP is transient and grows and shrinks as needed

– Error/Warning Log data – typically negligible and can be purged as needed

– Reporting data

• All reporting log rules off – 0 MB • Some or all reporting log rules are on

Typically data grows 100 MB – 3 GB per hour

Shrinks only when purged

But…

» Growth rate depends on page volume and which log rules are on

» Overall size depends on # days of data retained

– Audit Log data

• All audit log rules off – 0 MB

(14)

Tuning – InputAccel Database

Data Volume and Rates

• InputAccel Database transaction rates

Log Rules Enabled Log Rules Enabled Log Rules Enabled

Log Rules Enabled Estimated Transaction RateEstimated Transaction RateEstimated Transaction RateEstimated Transaction Rate

None IAS Tasks / Hour × 0.075

Reports only (no Audit) IAS Tasks / Hour × 2.5 Reports only (no Audit) IAS Tasks / Hour × 2.5 Audit only (no Reports) IAS Tasks / Hour × 5.0 Reports + Audit IAS Tasks / Hour × 7.5

(15)

Tuning – InputAccel Database

Improving Performance

• Defragment and rebuild indexes

– up_ReorganizeIndex – defragments all indexes – up_RebuildIndex – rebuilds all indexes

• Purge reporting and auditing data

– Reporting and Auditing tables grow continuously – Reporting and Auditing tables grow continuously – You must schedule purges via the Admin Console

• Recommendation: generate reports during non-peak hours

– Generating Reports runs complex queries that place a heavy load on

IADB and MSSQL

• Store MSSQL transaction logs and data files on separate hard

(16)

Tuning – Client Modules

• Each client module has its own unique tuning characteristics

– Example: val2xml slows as more IA Values are exported and is slower when triggered at

level 1 than level 7

– See “EMC® Captiva® InputAccel® and Dispatcher™ Version 6.5 Performance Sizing and

Tuning Guide” for details

• Parameters on the client machines can be modified to optimize performance

– Stored in settings.inilocated in %ALLUSERSPROFILE%\EMC\InputAccel – Stored in settings.inilocated in %ALLUSERSPROFILE%\EMC\InputAccel

– PrefetchDefault (default = 2) – the number of additional tasks the InputAccel Server sends to each client module

– FileCacheSize (default = 8)

– CacheSize(default = 1,048,576)

– CacheCount (default = 200,000), previously 20,000 – the number of files and IA Values

the client module caches

– IAClientDebug (default = 0) – set to 1 to capture the debug log iaclient.log in

(17)

Tuning – Client Modules

• CPU intensive client modules like NuanceOCR &

ImageEnhancement

– Require fast CPUs

– Run one instance for each CPU core.

• e.g. 4 CPU cores, run 4 instances of NuanceOCR

• Non-CPU intensive client modules like val2xml & Documentum • Non-CPU intensive client modules like val2xml & Documentum

Export

– Performance is limited by other resources (disk, network)

(18)

Tuning – Client Modules

Using InputAccel over a WAN…

• InputAccel Server, Database and Administration Console – Require high-speed, low-latency connections

– Must be on the same LAN as each other • Unattended client modules

– Often require high-speed, low-latency connections for best throughput – Generally do not need to be remote from the InputAccel Server

– Should be on the same LAN as the InputAccel Server • Attended client modules

– Performance varies by environment and module

– Detailed guidance is in the EMC® Captiva® InputAccel® and Dispatcher™

Version 6.5 Performance Sizing and Tuning Guide

(19)

Tuning – Client Modules

…Using InputAccel over a WAN

• Attended client module details

– ScanPlus

• Not recommended, but may perform acceptably with scanner hardware compression or small scanned images

• Works best when bandwidth is ≥50 Mbps and round trip latency ≤25 ms

– IndexPlus

• Generally performs well over a WAN (except for thumbnail display)

• Displaying the batch list takes about 25% longer • Displaying the batch list takes about 25% longer

• Works best when bandwidth is ≥1.5 Mbps and round-trip latency ≤50 ms

– Dispatcher Classification Edit and Dispatcher Validation

• Should not be used over a WAN

• Recommendation:Recommendation:Recommendation:Recommendation:consider using VMware View or Citrix for remote operators

– Module executes on the LAN, screen display is over the WAN – Supports remote scanning

• ScanPlus is on the LAN, the scanner is on the remote machine • Use scanner hardware compression for best results

– Maximizes InputAccel Server-to-client module throughput

(20)

Tuning – Capture Workflow

Trigger Levels

• Triggering at level 0 or 1

– Usually gives better throughput than level 7

– The tasks within a batch can be distributed among many client modules

– Which results in faster end-to-end processing of any single batch

• Triggering at level 7

– Is less work for InputAccel Server as it has fewer tasks to manage

– Under some circumstances may provide better overall throughput at the

expense that any single batch may take more time to process

• Unsupported: accessing external resources in IPP VBA code

– VBA execution with InputAccel Server is single-threaded

– The external resource may be slow or not present

– If InputAccel Server needs to wait for the resource, all other tasks block

– Put more complicated custom code logic on the client through the .NET Code

(21)

Diagnosing Performance Issues

• IA Server Performance Counter

– Batches loaded and loads/second

– Connections

– Disk bytes written & read/second

– VBA calls/second & queue length

– Processing Message Count

– Network bytes written & read/second

– Packets send & received/second

– Pending I/O (~ # of asynchronous sends in progress)

– Event (db) queue length

• Data Access Layer Performance Counters

– Data Requests/second

– % Load Factor

– Avg. Execution Time Millisec

(22)

EMC OnDemand

(23)

IIG Applications on EMC OnDemand

ENTERPRISE CAPTURE CONTENT MANAGEMENT CUSTOMER COMMUNICATIONS CASE MANAGEMENT Documentum Documentum xCP xCP Captiva

Captiva DocumentumDocumentum ECM

ECM DocumentDocumentSciencesSciences

INFORMATION GOVERNANCE SourceOne SourceOne Networks Networks Storage Storage

Virtualization and Security Virtualization and Security

Cloud Management Cloud Management

(24)

Summary/Key Takeaways

• Focus on system throughput, not per-task time

• Use high-speed hardware – disk drives and networks

• Minimize disk I/O where possible

– Keep active batches in memory – Avoid excessive reporting

– Avoid excessive reporting

• Parallelize – multi-core CPUs and task granularity

• Use performance counters to find bottlenecks

(25)

Q

&

A

Chris Lund

[email protected]

(26)

Learn More About EMC Captiva

(27)

Captiva @ Momentum 2011

Thursday

10:00 AM AP Automation: Best Practices for Capturing and

Integrating Paper Invoices into your Accounts Payable Processes

(28)

GET SOCIAL

DOWNLOAD DOWNLOAD DOWNLOAD DOWNLOAD Apple: Apple: Apple:

Apple: http://bit.ly/MMTM11http://bit.ly/MMTM11http://bit.ly/MMTM11http://bit.ly/MMTM11

Other: Other: Other:

(29)

(30)