SP Apps 1.1.4 Performance test
Test report
2012/10 Mai Au
SP Apps 1.1.0 Performance test ... 1
Test report ... 1
1. Purpose ... 3
2. Performance criteria ... 3
3. Environments used for performance testing ... 3
3.1. SP Apps servers ... 3
3.1.1 3 FEW (FrontEndWeb) servers ... 4
3.1.2 DB server ... 4
3.1.3 AD server, virtual machine ... 4
3.2. Tool for load testing ... 4
4. Verification data ... 4
5. Scenario ... 5
5.1. Scenario and script ... 5
5.2. Environment for running the scenario... 6
6. Result... 7
6.1. Result of ScaleBench ... 7
6.1.1. Usable user quantity ... 7
6.2. Analyzing server resource ... 8
6.2.1. Memory ... 8
6.2.2. Processor ... 11
6.2.3. Hard disk ... 13
1. Purpose
This performance test is to confirm the 3 following points in SP Apps 1.1.4. (1) Maximum user amount
(2) Consumed server resource at peak time (3) Comparing with SP Apps1.1.0
2. Performance criteria (1) Respond in 7 seconds
Response must be within 7 seconds since a request is sent
We consider the number of usage user when so many responses cannot be returned within 7 seconds the maximum user amount.
Because SP Apps operates on SharePoint 2010, we plus α to the 4 second rule and decides as 7 seconds.
(2)The rule of 50% usage ratio
It’s irrational that all registered users use SP Apps frequently, so we assume that 50% of the number of users sends 1 request/minute.
3. Environments used for performance testing 3.1. SP Apps servers
We performed the test on a system consisted of 2 FEW (Front End Web) servers, 1 DB server, 1 Active Directory.
We used the NLB (Windows Network Load Balancing Services) of Windows to cluster 2 FEW servers. Its settings are as follows:
Host properties
Cluster parameter Cluster manipulating mode Multicast
Port rule Port range Start: 0
Stop : 65535
Protocol TCP
Filter mode Multihost
Affinity:None
We must use None for Affinity because the requester must be from 1 same IP in Performance testing. ※Please refer to the URL for more details
3.1.1 2 FEW (Front End Web) servers
OS Windows Server 2008 R2 Enterprise Edition (x64) CPU
Intel® Xeon® CPU L5530 @ 2.40 GHz (2 プロセッサ)
Memory 48 GB
3.1.2 DB server
OS Windows Server 2008 R2 Enterprise Edition (x64) CPU
Intel® Xeon® CPU L5530 @ 2.39 GHz (2 プロセッサ)
Memory 48 GB
3.1.3 AD server, virtual machine
OS Windows Server 2008 Enterprise Edition (x64) CPU
Intel® Xeon® CPU L5530 @ 2.39 GHz (2 プロセッサ)
Memory 8.0 GB
3.2. Tool for load testing
Load testing tool Cybozu Scale Bench 1.1.0
OS Cent OS release 5.4
CPU Intel® Core™ i7-860 Processor @ 2.80GHz
Memory 6.00GB
4. Verification data
We used the following data for testing:
We used the tool to create application data. This tool was created by PG.
We assumed that we needed the data quantity accumulated on SPApps of 1000 users in 1 year. We assumed that there are 20 working days in a month, 8 hours a day.
User 1000 users
Department site Quantity of Site 10 sites
Discussion
Private Discussion: 600 items/1site 600*10 = 6000 items Public Discussion: 600 items/1site 600*10 = 6000 items
Scheduler 11 items/1site 11*10 = 110 items
Discussion
Private Discussion: 120 items/1site 120*50 = 1200 items
Public Discussion: 120 items/1site 120*50 = 1200 items
Scheduler 3 items/1site 3*50 = 150 items
Data amount in 1 year
Scheduler list (11*10 + 3*50)*365*5/7 = 67786
Discussion list 600*10 + 600*10 + 120*50 + 120*50 = 24000
5. Scenario
5.1. Scenario and script
We used the following scenario:
Script name Content Execution ratio
SP Apps_read 1. Login
2. Move to Top page of a site
3. Move to the Top page of Discussion 4. Click Discussion tag
5. Select any tag
6. Move to the Top page of Discussion 7. Click the Discussion tag
8. Select any item from the list 9. Move to the top page of the site 10. Move to the Top page of Scheduler 11. Open the Week view
12. Select any item from the list 13. Log out
72%
SP Apps_write 1. Log in
2. Move to Top page of a site
3. Move to the Top page of Discussion 4. Move to the [Add new] page of Discussion 5. Add a Discussion
6. Move to the Top page of Discussion 7. Move to the [Add new] page of Discussion 8. Add a Discussion
9. Move to the Top page of Scheduler 10. Move to the [Add new] page of Scheduler 11. Add an event
12. Log out
16%
2. Move to Top page of a site
3. Move to the Top page of Discussion 4. Click Discussion tag
5. Select any item from the list displayed 6. Post a comment to that Discussion 7. Move to the Top page of Discussion 8. Click Discussion tag
9. Select any item from the list displayed 10. Post a comment to that Discussion 11. Move to the Top page of Scheduler 12. Open the Week view
13. Select any item from the list displayed 14. Post a comment to that event
15. Log out
These scenario and scripts are saved in the following location: ScaleBench Script used for SP Apps1.1.4 performance test
http://svn.dev.cybozu.co.jp/svn/qa/SPApps/orion/patch/1.1.4/scenario/
5.2. Environment for running the scenario As follows
grinder.properties
grinder.threads （VU quantity for each server） 1000 grinder.duration (Duration of performance test (milisec)) 2400000 scalebench.properties
scalebench.wait_mean (Average think time (milisec)) 60000 scalebench.wait_range (Difference of think time (milisec)） 30000 scalebench.peak_at (The percentage of duration at which
VU quantity becomes maximum) 0.8
Access user:
500 users access into 10 Department sites 500 users access into 50 Project sites
6. Result
6.1. Result of ScaleBench 6.1.1. Usable user quantity
As follows
Pattern VU quantity by the time
the average response time exceeds 7 seconds
Usable user quantity
SP Apps 1.1.0 293 586
SP Apps 1.1.4 1000 2000
The usable user quantity in SPApps1.1.4 is 2000 users. ・SP Apps 1.1.4(1000VU)
Although there are 2 operations (create private discussion and post comment for public discussion) has response time over 7s, PG had investigated but he didn’t found any part can be tweaked more.
6.2. Analyzing server resource
I used the Performance Counter of Windows on the verification server to output the server resource each 5 seconds. The output targets are as below:
Object Counter Content Bottleneck and scale
for judgment
Memory Available Bytes Usable memory capacity Below 5% of entire memory
Pages/sec The number of Paging in each second
Over 1000 Processor(_Total) % Processor
Time
The percentage of non-idle time of processor
Over 85% % User Time The percentage of the user process
running time of processor
% Idle Time The percentage of idle of time of processor
Physical Disk % Disk Time The percentage of the time when disk is being read or written
Over 80% Avg. Disk
Queue Length
The unprocessed request quantity waiting for disk writing and reading to finish
Over spindle amount × 2
System Processor Queue Length
The waiting thread quantity in processor
Over CPU quantity × 2
Reference: Windows Management : Observing server performance- Microsoft TechNet
http://207.46.16.252/ja-jp/magazine/2008.08.pulse.aspx
Because there was little difference in the resource usage of 2 FEW servers, I explained only one of them.
6.2.1. Memory
The usable memory capacity is as below:
After 32 minutes, which is the peak time of SP Apps, there was no big fluctuation in memory usage amount in FEW servers and DB server.
・SP Apps 1.1.4_1000VU (1 FEW server)
Here is the paging quantity in each second:
Although there was a sudden rise in paging quantity, there was no excess paging that could cause memory’s bottleneck.
・SP Apps 1.1.4_1000VU (1 FEW server )
6.2.2. Processor
After 32 minutes, which is the peak time of SP Apps, the processor of DB server and Web server has a percentage of usage time about 50%. Bottleneck is not happened at DB and Web servers
・SP Apps 1.1.4_1000VU (1 FEW server)
Here is the thread quantity that was waiting for process in processor.
Although there was a sudden rise, this CPU bottleneck state didn’t last long. ・SP Apps 1.1.4_1000VU (1 FEW SERVER)
6.2.3. Hard disk
Here is the percentage of disk usage time. Although there was a sudden rise, the time was not long enough to cause bottleneck.
・SP Apps 1.1.4_1000VU (1 FEW SERVER)
7. Conclusion
(1) Maximum user amount
The usable user quantity of SP Apps 1.1.4 is 2000. (2) Consumed server resource at peak time
Object Counter Consumed resource at peak time
DB server Web server
Memory Available Bytes 14.6% of entire
memory
8.3% of entire memory
Pages/sec 0 28.7
Processor(_Total) % Processor Time < 40% ~ 48%
% User Time ~ 20% ~ 43%
% Idle Time ~ 60% ~ 52%
Physical Disk % Disk Time < 5% < 5%
Avg. Disk Queue Length
System Processor Queue Length < 5 < 5
(3) Comparing with SP Apps1.1.0
Pattern VU quantity by the time
the average response time exceeds 7 seconds
Usable user quantity
SP Apps 1.1.0 293 586
