Jmeter was the tool considered to proceed with the assessment of the QoS relevant metrics, for both, the generated code and the case study application. Two services were chosen to be analyzed,
93
given the similarity between the implementations in both cases, and being two of the most used services in the case study application:
1. Search food:
o GET nutrition.clinic.gateway/api/foods ?filter=name:EQ:inhame o GET mypocketnutritionist/api/v1/tcas/search string=inhame
2. Create client:
o POST nutrition.clinic.gateway/api/users o POST mypocketnutritionist/api/v1/registration
5.3.1 Collected data
The tool selected allows to simulate multiple tests scenarios, by choosing the number of users (threads), time conditions (interval between requests, ramp-up time, etc.), variables that will be used in the requests, etc. For the current analysis a total of 50 requests were made, and the following QoS metrics registered, latency (ms), response time (ms) and throughput (number of requests per second).
5.3.1.1Statistic summary
Table 24 and Table 25 show some descriptive statistics associated with each service analyzed and for each implementation considered.
Table 24 - Search food service QoS descriptive statistics (N=50).
QOS
Generated code Case study
Min. 1st
Qu. Median Mean 3rd Qu. Max. Min. Qu. 1st Median Mean 3rd Qu. Max.
Latency, ms 5 5 5 5.7 6 15 7 7.25 8 7.78 8 9
Response
time, ms 5 5 5 5.7 6 15 7 7.25 8 7.78 8 9
Throughput,
nº requests/s 10.2 9.6
Table 25 - Create user service QoS descriptive statistics (N=50).
QOS
Generated code Case study
Min. 1st
Qu. Median Mean 3rd Qu. Max. Min. Qu. 1st Median Mean 3rd Qu. Max.
Latency, ms 5 7 7 7.16 7 20 7 10 11 10.32 11 13
Response
time, ms 5 7 7 7.16 7 20 7 10 11 10.32 11 13
Throughput,
94
Reading the obtained statistics for each service, and comparing the overall performance of the targeted implementations, at first sight it could be concluded that the generated solution provides a faster response time for both services. However, it is important to state that the execution time (Response time - Latency) is zero for both cases, which indicates that the time spent executing the code can be neglected and the difference registered is probably associated with other variables. Since both the implementations use a MySQL server as SGBD, the deployment servers where the solutions are deployed could be the origin of the registered differences: Glassfish in the generated code and Redis for the case study application.
This could indicate that, despite the original suspicions that the poor performance of some services in the case study application was related to the code implementation, the problem actually resides in the technological stack used in the implementation.
The aforementioned conclusions impact what can be inferred from the planned hypotheses, since they might not be related to the implementation itself.
5.3.1.2Hypothesis analyses
As previous mentioned the hypotheses in analysis regarding the quality of service metrics for the services in analysis are:
• H03: No significant difference exists between the web services performance of the generated code and the case study project.
• H13: The generated web services performance is relatively better that the one provided by the case study project.
The approach follows the steps referenced in the previous section hypotheses analysis, by start testing the distribution normality through the Shapiro-Wilk test - Table 26 and Table 27.
Table 26 - Shapiro-Wilk test of normality: search service.
Generated code Case study
Metric p-value H0 p-value H0
Latency 8.215e-13 rejected 2.869e-09 rejected
Response time 8.215e-13 rejected 2.869e-09 rejected
Table 27 - Shapiro-Wilk test of normality: create service
Generated code Case study
Metric p-value H0 p-value H0
Latency 5.551e-13 rejected 3.739e-06 rejected
Response time 5.551e-13 rejected 3.739e-06 rejected
Once again, the statistical evidences suggest that the data collected from both services do not follow a normal distribution. This leads the hypotheses evaluation through non-parametric tests.
The analysis conducted follows the same steps that the one in Section 5.2.2.2. First the distribution shape - Figure 30 - is examined for its similarity. Then the Mann-Whitney-Wilcoxon tests are executed, for each service, and each implementation - Table 28 and Table 29.
95 Search service latency Search service response time
Create service latency Create service response time Figure 30 - Shape distributions for metrics values.
Table 28 - Mann-Whitney-Wilcoxon test: search service.
Metric p-value H02 p-value H12
Latency 2.2e-16 accepted 2.2e-16 accepted
Response time 2.2e-16 accepted 2.2e-16 accepted
Table 29 - Mann-Whitney-Wilcoxon test: create service.
Metric p-value H02 p-value H12
Latency 2.783e-16 accepted 2.2e-16 accepted
Response time 2.783e-16 accepted 2.2e-16 accepted
The conclusions that can drawn from the results attained are identical to the ones obtained before, in the comparison with the Apimatic solution. It can be said that statistical evidences advocate that both solutions do not show relevant differences in the overall performance of the web services QoS (H0). When executing the unilateral version of the test, the results suggest that the QoS for the generated solution performs relatively better than the case study.
97
Chapter 6
Conclusion and future work
This chapter aims to summarize the results of the work done, highlighting the completed objectives, difficulties encountered, limitations and future work. Finally, it concludes with some statements about the work carried out in the framework of the TMDEI course.