Testing of inter-process communication and synchronization of ITP LoadBalancer software via model-checking

(1)

Testing of inter-process communication and

synchronization of ITP LoadBalancer software via

model-checking

Marko van Eekelen, Stefan ten Hoedt, Ren´e Schreurs, Yaroslav S. Usenko

Aia & LaQuSo

(2)

Outline

Aia and the Case Study

Case Analysis and Reverse-Engineering

Modeling and Analysis with the mCRL2 Toolset

(3)

Aia Software BV

I Leading producer of high capability Business Document Output

Solutions

I Established 1988

I Expanding partner network

I 800 customers in>25 countries

I Corporate Headquarters - Nijmegen, The Netherlands

I Regional Centers I _Benelux I _Germany I _UK I _{North America} I _Switzerland I _Australia

(4)

Applications of ITP

I Insurance I _Policies I _Endorsements I _Renewals I Financial Services I _Statements I _{Correspondence} I _Contracts I Government I Taxation I _Permits I _{Correspondence} I Independent Software

(5)

Applications of ITP

I Insurance I _Policies I _Endorsements I _Renewals I Financial Services I _Statements I _{Correspondence} I _Contracts I Government I Taxation I _Permits I _{Correspondence} I Independent Software Vendors

(6)

Applications of ITP

I Insurance I _Policies I _Endorsements I _Renewals I Financial Services I _Statements I _{Correspondence} I _Contracts I Government I Taxation I _Permits I _{Correspondence} I Independent Software

(7)

Applications of ITP

I Insurance I _Policies I _Endorsements I _Renewals I Financial Services I _Statements I _{Correspondence} I _Contracts I Government I Taxation I _Permits I _{Correspondence} I Independent Software Vendors

(8)

(9)

Intelligent Text Processing (ITP) and its Load-Balancer

Client 1 Server 2 Server 3 Client 3 Client 2 Server 1 Load Balancer Intranet (TCP) Internet (TCP)

(10)

Issues

I LoadBalancer does not respond at all (deadlocks)

I Free workers are not used (partial deadlocks)

(11)

Artifacts

I source code in C for windows (7681 lines)

I application layer protocol documentation

I verbal information during meetings, phone and e-mail

communication. I Threads I MutExes I WSA I WaitForMultipleObjects I CallBack functions.

(12)

Artifacts

I source code in C for windows (7681 lines)

I application layer protocol documentation

I verbal information during meetings, phone and e-mail

communication. I Threads I MutExes I WSA I WaitForMultipleObjects I CallBack functions.

(13)

A Typical Use Case Scenario

Client object Client

request to print

work?

yes! (partners)

wake−up : get data

get data

data

wake−up : data ready

process data work?

Server Server object

(14)

Reverse Engineering of the LoadBalancer Design

Socket Administration (OS) Socket Administration (OS) C2 C1 C3 S1 S2 request partner partner partner wake−up event wake−up event wake−up event socket event socket event socket event socket socket event event

(15)

Properties to Check

I deadlock freedom

I critical logs

I if the partner ofAisB>0, then the partner ofBisAor0.

I a server may not sleep w/o a partner (except when a request is

pending to it)

I limits on locking

(16)

mCRL2 Language

mCRL2 is based on process algebra (ACP) and algebraic (equational) data types. Specification structure:

I data types definitions (sort,func,map,rew)

I actions and communication functions definitions (act,comm)

I process definitions (proc): equations involving:

p ::= a(→−t) | δ | _Y₍→−_t₎ | _p+_p | _p·_p | _pk_p | _c→_p_p | _c→

p | P

d:Dp | τI(p) | ∂H(p) | ρR(p) | ∇G(p) | ΓC(p)

I initial state (init).

Extensions to process algebra:

I action parameterized by data (a(d)|_b₍_e₎ ≈ ₍_d=_e₎→_c₍_d₎_),

I P_d_:_Dpandc→_x_y

(17)

The mCRL2 Toolset (www.mcrl2.org)

SVC formula checking Specification Text optimization xsim sim lpe2lts lpe2pbes

model checking simulation

visualisation FLTS LPE mCRL2 ATerm model checking lpeparelm lpedataelm lpesumelm

evaluator(CADP) ltspp + dot (graphviz)

(state−space generation, deadlock checking, action search, confluence reduction) ltsmin ltsupdate (comparison, minimization) (renaming of actions) mcrl22lpe lpepp lpe2torx lpestructelm lpeupdate lpeinvelm lpeconfelm lperewr lpeconstelm

formcheck(eq−BDD−based prover) confcheck

(18)

Experiments

Experiments on a 3Ghz 32 bit machine with 4Gb RAM.

#clients #servers time #levels #states #transitions

1 1 7m 38s 241 657k 1.38M 1 2 3h 01m 367 18M 38.5M 2 1 9h 55m 444 54M 141M 1 3 13h∗ 481 213M 465.5M 2 2 >113h∗ >215 >511M >1121M ∗

(19)

Detected Issues

I partner links inconsistent

I _{set partner to 0 was forgotten for one of the parties} I _{found by model-code comparison}

I _{confirmed to be a problem by model-checking} I server sleeping w/o a partner

1 set client’s partner link to 0 before waking up the server 2 forgotten to wake up the server

I _{1st found by model-code comparison, 2nd by model-checking}

I critical logs could occur

1 sending request for disconnect to itself happened in a wrong state (forgot to change the state)

2 request to wake up can lead to an inappropriate state change when server disconnects (not critical)

I number of requests exceeds the limit

(20)

Detected Issues

I _{confirmed to be a problem by model-checking}

I server sleeping w/o a partner

I _{1st found by model-code comparison, 2nd by model-checking} I critical logs could occur

(21)

Detected Issues

I number of requests exceeds the limit

(22)

Detected Issues

(23)

A Faulty Scenario

Client object Client request to print work? yes! (partners)

wake−up : get data

get data

work?

Server Server object

disconnect

(24)

Conclusions and Future Work

Conclusions

I Session layer of Load Balancer is modelled

I A number of properties are verified

I Number of issues discovered, communicated and corrected

I Cases up to 1 client and 3 servers and 2 clients and 1 server were fully analyzed

I Case with 2 clients and 2 servers was partially analyzed

I Modification of the model and further analysis are possible

(25)

Conclusions

(26)

Conclusions

(27)

Conclusions

I Cases up to 1 client and 3 servers and 2 clients and 1 server were

fully analyzed

(28)

Conclusions

fully analyzed

(29)

Conclusions

fully analyzed

(30)

Conclusions

fully analyzed

(31)

Conclusions

fully analyzed

(32)

Future Work

I Certification of the LoadBalancer software

I For the session layer: more properties

(33)

Future Work

(34)