ON THE ROAD TO BENCHMARKING BPMN 2.0 WORKFLOW ENGINES

(1)

Vincenzo Ferme

, Cesare Pautasso

Faculty of Informatics

University of Lugano (USI)

Switzerland

Marigianna Skouradaki, Dieter H. Roller, Frank Leymann

Institute of Architecture and Application Systems

University of Stuttgart

Germany

ON THE ROAD TO

BENCHMARKING BPMN 2.0

WORKFLOW ENGINES

(2)

Application Server

DBMS

2

Workflow Engine

What is a Workflow Engine?

Job

Executor

Engine

Core

Transaction Manager

Instance

Database

Persistent Manager

Process Navigator

A

B

C

D

Task Dispatcher

Users

Service Invoker

…

Web

Service

(3)

3

Many Business Process Modeling/Execution Languages

BPEL

EPC

XPDL

YAWL

BPMN

1992

1998

2002

2004

PNML

2008

…

(4)

4

BPMN 2.0: A Widely Adopted Standard

Jan 2011

BPMN 2.0

Jan 2014

BPMN 2.0.2

https://en.wikipedia.org/wiki/List_of_BPMN_2.0_engines

ISO/IEC 19510

(5)

5

Why do we need a benchmark?

(6)

5

companies, developers

1. How to choose the best engine according to

the company requirements?

2. How to choose the best engine

according to the company business

process models?

B

C

(7)

5

companies, developers

1. How to choose the best engine according to

the company requirements?

2. How to choose the best engine

according to the company business

process models?

B

C

D

4. How to find out the engine bottlenecks?

3. How to evaluate performance improvements

(8)

6

(9)

6

Main Challenges in Benchmarking BPMN 2.0 Workflow Engines

WORKLOAD CHARACTERIZATION

(10)

6

WORKLOAD CHARACTERIZATION

BENCHMARK EXECUTION

2. Define the Load Functions

1. Define the Workload Mix

20%

80%

D

A

B

C

(11)

6

WORKLOAD CHARACTERIZATION

BENCHMARK EXECUTION

4. Asynchronous execution of business processes

Instance

Database

5. Define meaningful and reliable KPIs

3. Deal with engine-specific interfaces

and BPMN 2.0 customizations

x

Eng

ine

↔

U

se

rs

C

lie

nt

→

E

ng

ine

Eng ine ↔ W eb S erv ic es

20%

80%

D

A

B

C

(12)

7

1. Define the Workload Mix

(13)

7

Data Flow

A

B

C

B

A

Control Flow

Events

Activities

Task Types

Execution Behavior

A

B

C

D

E

F

G

H

I

(14)

8

NUMBER OF ENGINES SUPPORTING THE FEATURE

N

U

MB

ER

O

F

RE

A

L-W

O

RL

D

MO

D

EL

S

(15)

8

N

U

MB

ER

O

F

RE

A

L-W

O

RL

D

MO

D

EL

S

0

200

(16)

8

1. Define the Workload Mix

8

N

U

MB

ER

O

F

RE

A

L-W

O

RL

D

MO

D

EL

S

0

200

12

950

(17)

8

N

U

MB

ER

O

F

RE

A

L-W

O

RL

D

MO

D

EL

S

0

200

12

950

2

4

5

8

10

F

400

600

700

800

(18)

9

Workflow

Engine

Users

A

B

C

D

Start

Web Services

Users

Events

Instance

Database

Application Server

DBMS

Web

Service

(19)

10

3. Deal with engine-specific interfaces

and BPMN 2.0 customizations

Workflow

Engine

Users

Loading Driver

A

B

C

D

Instance

Database

Application Server

DBMS

Web

Service

(20)

11

4. Asynchronous execution of processes

Workflow

Engine

Users

A

B

C

D

Start

Loading Driver

Instance

Database

Application Server

DBMS

Web

Service

(21)

11

4. Asynchronous execution of processes

Workflow

Engine

Users

A

B

C

D

Start

Loading Driver

Instance

Database

End

Application Server

DBMS

Web

Service

(22)

12

The BenchFlow Project

Design the first benchmark to assess and

compare the performance of Workflow Engines

that are compliant with Business Process Model

and Notation 2.0 (BPMN 2.0) standard

“

”

Eng

ine

↔

U

se

rs

Instance

Database

C

lie

nt

→

E

ng

ine

Eng ine ↔ W eb S erv ic es

20%

80%

D

A

B

C

(23)

13

What we need:

even more (anonymized) real-world BPMN 2.0 process models

A

B

C

D

H

F

E

G

REAL-WORLD

PROCESSES

(24)

13

What we need:

even more (anonymized) real-world BPMN 2.0 process models

A

B

C

D

H

F

E

G

REAL-WORLD

PROCESSES

[SOSE2015]

a

3

a

2

a

1

a

6

a

5

a

4

Graph

Matching

REOCCURRING STRUCTURES

Skouradaki et al.

(25)

Selection

Criteria

13

What we need:

A

B

C

D

H

F

E

G

REAL-WORLD

PROCESSES

[SOSE2015]

a

3

a

2

a

1

a

6

a

5

a

4

Graph

Matching

Skouradaki et al.

(26)

Selection

Criteria

13

What we need:

A

B

C

D

H

F

E

G

REAL-WORLD

PROCESSES

WORKLOAD MIX

a

1

a

2

a

1

a

3

a

5

50%

Composition

Criteria

[SOSE2015]

a

3

a

2

a

1

a

6

a

5

a

4

Graph

Matching

Skouradaki et al.

(27)

14

Enabling the Benchmark Execution

Workflow

Engine

Application Server

Users

DBMS

A

B

C

D

Loading Driver

Instance

Database

Web

Service

(28)

15

Workflow

Engine

Instance

Database

Users

Faban

harness

Faban Drivers

A

B

C

D

Loading

Functions

Application Server

DBMS

Web

Service

(29)

Workflow

Engine

16

harness

Web

Service

DBMS

1. Flexible deployment

2. Flexible HW Resources

3. Frozen Initial Condition

Faban Drivers

Faban

Docker

Containers

Servers

A

B

C

D

Loading

Functions

(30)

17

harness

Faban Drivers

Faban

+

Workflow

Engine

(31)

1.

Automatically deploy and start the benchmark environment;

17

harness

Faban Drivers

Faban

+

Workflow

Engine

(32)

1.

17

harness

Faban Drivers

Faban

+

Workflow

Engine

DBMS

A

B

C

D

(33)

1.

17

harness

Faban Drivers

3.

Determine when the benchmark ends;

Faban

+

Workflow

Engine

DBMS

A

B

C

D

MONITOR

(34)

1.

17

harness

Faban Drivers

3.

Determine when the benchmark ends;

Faban

+

Workflow

Engine

DBMS

A

B

C

D

MONITOR

Instance

Database

CO

LL

EC

T

O

RS

2.

Automatically deploy the workload mix;

(35)

18

The BenchFlow Project Next Steps

•

Release the first prototype of the Benchmark environment

»

Yes:

Abstract the Interaction with the Engines; Automatic Deploy and

Undeploy of the S.U.T.; Execution and Process Log Gathering

»

No:

Automatic Generation of Drivers; Users, Web Services and External

Catching Business Events

•

Release the first prototype of the Workload Mix synthesizer

•

First Experiments with KPIs Definition and Computation

•

Collect More Process Models and Process Execution Logs

(36)

BACKUP SLIDES

Vincenzo Ferme

, Cesare Pautasso

Faculty of Informatics

University of Lugano (USI)

Switzerland

Marigianna Skouradaki, Dieter H. Roller, Frank Leymann

Institute of Architecture and Application Systems

University of Stuttgart

Germany

• Cited Works;

• Related Works.

(37)

20

Cited Works

DBMS

[SOSE2015]

Skouradaki, Marigianna; Goerlach, Katharina; Hahn, Michael; Leymann, Frank.

Application

of Sub-Graph Isomorphism to Extract Reoccurring Structures from BPMN 2.0 Process

Models

. In Proceedings of 9th International IEEE Symposium on Service-Oriented System

(38)

21

Related Works

DBMS

Active Endpoints Inc.

Assessing ActiveVOS performance

, 2011. http://www.activevos.com/

content/developers/ technical_notes/assessing_activevos_performance.pdf.

D. Bianculli, W. Binder, and M. L. Drago.

SOABench: Performance evaluation of service-oriented

middleware made easy

. In Proc. of ICSE’10 - Volume 2, pages 301–302, 2010.

J. Cardoso.

Business process control-flow complexity: Metric, evaluation, and validation

.

International Journal of Web Services Research, 5(2):49–76, 2008.

G. Din, K.-P. Eckert, and I. Schieferdecker.

A workload model for benchmarking BPEL engines.

In

Proc. of ICSTW’08, pages 356–360, 2008.

M. Dumas, L. Gar

ć

ıa-Bañuelos, and R. M. Dijkman.

Similarity search of business process models

.

IEEE Data Eng. Bull., 32(3):23–28, 2009.

J. Gray.

The Benchmark Handbook for Database and Transaction Systems.

Morgan Kaufmann,

2nd edition, 1992.

G. Hackmann, M. Haitjema, C. Gill, and G.-C. Roman.

Sliver: A BPEL workflow process execution

engine for mobile devices

. In Proc. of ICSOC’06, pages 503–508. Springer, 2006.

S. Harrer, J. Lenhard, and G. Wirtz.

BPEL conformance in open source engines

. In Proc. of

(39)

22

Related Works

DBMS

K. Huppler.

The art of building a good benchmark

. In Performance Evaluation and Benchmarking,

pages 18–30. Springer, 2009.

Intel and Cape Clear.

BPEL scalability and performance testing

. White paper, 2007.

F. Leymann.

Managing business processes via workflow technology.

In Proc. of VLDB 2001, pages

729–, 2001.

A. Liu, Q. Li, L. Huang, and M. Xiao.

Facts: A framework for fault-tolerant composition of

transactional web services

. IEEE Trans. on Services Computing, 3(1):46–59, 2010.

J. Mendling.

Metrics for Process Models: Empirical Foundations of Verification, Error

Prediction, and Guidelines for Correctness

. Springer, 2008.

I. Molyneaux.

The Art of Application Performance Testing: Help for Programmers and Quality

Assurance

. O’Reilly, 2009.

M. Z. Muehlen and J. Recker.

How much language is enough? theoretical and practical use of the

business process modeling notation

. In Proc. of CAiSE’08, pages 465–479, 2008.

C. Röck and S. Harrer.

Literature survey of performance benchmarking approaches of BPEL

engines

. Technical report, Otto-Friedrich University of Bamberg, 2014.

D. H. Roller.

Throughput Improvements for BPEL Engines: Implementation Techniques and

(40)

23

Related Works

DBMS

N. Russell, W. M. van der Aalst, and A. Hofstede.

All that glitters is not gold: Selecting the right

tool for your BPM needs

. Cutter IT Journal, 20(11):31–38, 2007.

D. Schumm, D. Karastoyanova, O. Kopp, F. Leymann, M. Sonntag, and S. Strauch.

Process fragment

libraries for easier and faster development of process-based applications

. CSSI, 2(1):39–55,

2011.

M. Skouradaki, D. Roller, C. Pautasso, and F. Leymann.

BPELanon: Anonymizing BPEL processes

. In

Proc. of ZEUS’14, pages 9–15, 2014.

Sun Microsystems.

Benchmarking BPEL service engine

, 2007. http://wiki.open-esb.java.net/

Wiki.jsp?page=BpelPerformance.html.

B. Wetzstein, P. Leitner, F. Rosenberg, I. Brandic, S. Dustdar, and F. Leymann.

Monitoring and

analyzing influential factors of business process performance

. In Proc. of EDOC’09, pages 141–

http://design.inf.usi.ch/research/projects/benchflow

Sun Microsystems.