Dissertation Masters in Informatics Engineering

(1)

Dissertation

Masters in Informatics Engineering

Evaluating

Web Services Security

Nuno Manuel dos Santos Antunes

[email protected]

Advisor:

Prof. Marco Vieira

10-07-2009

Department of Informatics Engineering

Faculty of Science and Technology

UNIVERSITY OF COIMBRA

(2)

(3)

3

Abstract

Although web services are becoming business-critical components, they are often deployed with critical software bugs, causing security vulnerabilities that can be maliciously exploited. Develop time constraints and developers not specialized in security often lead to security cautions being disregarded, giving utmost importance to t the use of automated security testing tools to detect existing security vulnerabilities. However, automated security testing tools often do not deserve the confidence that developing teams put on them. In fact, previous research shows that many vulnerabilities remain undetected even when using well-known and widely used vulnerability detection tools.

The present work has two main contributions: the evaluation of existing tools and the proposal of new approaches for the detection of vulnerabilities. First we evaluate existing tools to assess their effectiveness in the detection of vulnerabilities in web services environments. Results show that many web services are deployed with security vulnerabilities (being SQL Injection the most common type of vulnerability in this context) and that security test tools present an unsatisfactory effectiveness in web services environment (low coverage and high number of false positives). This way, we propose two new techniques for detection of security vulnerabilities in web services. The first is based on penetration testing and target SQL Injection vulnerabilities. The second is a gray-box approach for the detection of SQL Injection and XPath Injection vulnerabilities.

The experimental evaluation shown that the penetration testing tool achieved higher effectiveness than the web security scanners on detecting SQL Injection vulnerabilities, showing that is possible to develop a vulnerability scanner for web services that performs much better than the commercial ones currently available. In relation to the proposed gray-box approach, experimental evaluation has shown that it performs much better than known tools (including commercial ones), achieving extremely high detection coverage while maintaining the false positives rate very low.

Keywords

Web Services, Security, Service Oriented Architecture, Software testing, SQL Injection, XPath Injection, Software vulnerabilities, Vulnerability detection, Dependability, black-box testing, gray-box testing.

(4)

(5)

5

Acknowledgments

I would like to start by thanking to Professor Marco Vieira as this work would never be possible without his orientation. Marco has revealed to be an excellent advisor being his motivation and, very important, patience a fundamental piece to my evolution during this work. The high level quality and productivity that “demand” from his students turn the work in a very hard challenge, but even more rewarding.

Also, I thank to Professor Henrique Madeira for his contribution during my research work. I also thank Professor Henrique and Professor Edmundo Monteiro who, as members of the jury, helped to improve the quality of this work with their constructive comments during the presentation of dissertation plan.

I also thank the good humour, suggestions and examples of my laboratory colleagues. To my friends, I thank for their support.

I would like to express my gratitude to my family, especially to my parents for the examples of hard work they gave to me and to my brother José Carlos that always believed in my capabilities and always encouraged me to pursuit a life of success. Finally, I want to thank Marina, for helping and supporting me in work and in life. For share with me happiness and sadness, and for never let me give up on the pursuit of my goals.

(6)

(7)

7

List of Publications

This thesis is based on work presented in the following papers1:

1. Antunes, N. and Vieira, M., “Comparing the Effectiveness of Penetration Testing and Static Code Analysis on the Detection of SQL Injection Vulnerabilities in Web Services”, IEEE 15th Pacific Rim International Symposium on Dependable Computing (PRDC’09), Shanghai, China, November 2009

2. Antunes, N. and Laranjeiro, N. and Vieira, M. and Madeira, H., “Effective Detection of SQL/XPath Injection Vulnerabilities in Web Services”, IEEE International Conference on Services Computing (SCC 2009), Bangalore, India, September 2009

3. Antunes, N. and Vieira, M., “Detecting SQL Injection Vulnerabilities in Web Services”, Fourth Latin-American Symposium on Dependable Computing (LADC 2009), João Pessoa, Paraíba, Brazil, September 2009

4. Vieira, M. and Antunes, N. and Madeira, H., “Using Web Security Scanners to Detect Vulnerabilities in Web Services”, 39th Annual IEEE/IFIP International Conference on Dependable Systems and Networks (DSN 2009), Estoril, Lisbon, Portugal, June 2009

1

(8)

(9)

9

List of Figures

Figure 1. Web Services Typical Structure. ... 17

Figure 2 - Application errors detected. ... 29

Figure 3 – SQL Injection vulnerabilities. ... 30

Figure 4 - False Positives and Doubtful vulnerabilities. ... 32

Figure 5 - SQL Injection vulnerabilities without False Positives. ... 33

Figure 6 - Vulnerabilities distributed per type. ... 35

Figure 7. Configuration for penetration tests... 39

Figure 8. Execution profile for Phase 2. ... 41

Figure 9. Web service response analysis. ... 43

Figure 10. SQL Injection vulnerabilities reported. ... 45

Figure 11. SQL Injection vulnerabilities classified. ... 46

Figure 12. SQL Injection vulnerabilities in TPC-App. ... 48

Figure 13. Steps for the detection of vulnerabilities. ... 51

Figure 14. The AOP-based configuration... 52

Figure 15. Example of SQL commands execution. ... 53

Figure 16. Execution profile for Phase 2. ... 54

Figure 17. Results for penetration testing. ... 59

Figure 18. Intersections for penetration testing. ... 60

Figure 19. Results for static code analysis. ... 61

(11)

11

List of Tables

Table 1 - Overall results table... 28

Table 2 - False Positives and Doubtful vulnerability percentages. ... 32

Table 3 - False positives for other vulnerability types. ... 33

Table 4 - Coverage for SQL Injection. ... 34

Table 5. Examples of SQL Injection attack types. ... 40

Table 6. Rules for the analysis of opposite responses. ... 43

Table 7. Vulnerabilities detection coverage. ... 47

Table 8. Examples of problematic vulnerabilities. ... 50

Table 9. Web services characterization. ... 56

(12)

(13)

13

Chapter 1 Introduction

Ranging from on-line stores to large corporations, web services are increasingly becoming a strategic vehicle for data exchange and content distribution [13]. Web services provide a simple interface between a provider and a consumer [13] and are supported by a complex software infrastructure, which typically includes an application server, the operating system and a set of external systems (e.g., databases, payment gateways, etc). The Simple Object Access Protocol (SOAP) [13] is used for exchanging XML-based messages between the consumer and the provider over the network (using for example http or https protocols). A web service is described using WSDL (Web Services Definition Language) [14]. WSDL is a XML format used to generate server and client code and to specify the format of the messages to be exchanged between server and client. A broker or registry is often used to enable applications to find web services.

As any web application, web services are so widely exposed that any existing security vulnerability will most probably be uncovered and exploited by hackers. Moreover, hackers are moving their focus to applications’ code, often improperly implemented, searching for vulnerabilities by exploring applications’ inputs with specially tampered values. These values can take advantage of existing vulnerabilities representing considerable danger to the application’s owner, for instance, by giving to an attacker access to reserved resources to read, modify or destroy. A very important point is that, normally, traditional security mechanisms (Firewalls, Intrusion Detection Systems, Encryption, etc) cannot mitigate these attacks.

Security vulnerabilities like SQL Injection and XPath Injection, seem particularly relevant in web services as they are directly related to the way the web service code is structured [44][45] and also because services often access sensitive organizational data. Basically, these types of attacks take advantage of improper coded applications to change SQL commands that are sent to the database or tamper XPath queries used to access parts of an XML document. As web services are usually a part of business-critical systems these vulnerabilities represent considerable danger to the organization. Many published studies and reports show that, in general, web applications often present dangerous security flaws [2][33]. To prevent vulnerabilities, developers should apply best coding practices, perform security reviews of the code, execute penetration tests, use code vulnerability analyzers, etc. However, many times developers focus on the implementation of functionalities and on satisfying the costumer’s requirements and disregard security aspects. Additionally, numerous developers are not specialized on security and the common time-to-market constraints limit an in-depth test for security vulnerabilities. There are three main approaches to find vulnerabilities in web applications:

(14)

14

 Black-box testing: consists in the analysis of the execution of the application in search for vulnerabilities. This approach is also known as penetration testing.

 White-box testing: consists in the analysis of the source code (or bytecode) of the application to identify vulnerabilities. This approach is also known as static code analysis.

 Gray-box testing: combine the two previous approaches. This can be done in very different ways.

In web services environments, automated black-box testing tools have a very important role because they can be used both by providers and consumers. These tools are named web security scanners (or web vulnerability scanners) and are usually seen as a practical way to disclose vulnerabilities in web applications as they provide an easy way to do the search automatically. Nowadays some of these scanners support security testing in web services. Most of these scanners are commercial tools (e.g., and HP WebInspect [21], IBM Rational AppScan [23], Acunetix Web Vulnerability Scanner [3]) but there are also some free application scanners (e.g., Foundstone WSDigger [19] and WSFuzzer [55]) that usually lack most of the functionalities of their commercial counterparts. Previous research suggests that the effectiveness of scanners in the detection of vulnerabilities varies a lot [17]. To confirm this in the context of web services, we have used four well know vulnerability scanners to identify security flaws in 300 public web services. The goal was to study the effectiveness of the scanners and to try to identify common types of vulnerabilities in web services environments. A large number of vulnerabilities have been observed, which confirms that many services are deployed without proper security testing. Additionally, the differences in the vulnerabilities detected and the high number of false-positives, and low coverage observed highlight the limitations of web vulnerability scanners on detecting security vulnerabilities in web services.2

These results show that vulnerability scanners are not an effective way to perform a web services security evaluation and suggested an open field to develop tools with an effective performance on the detection of security flaws in web services. This way, a second contribution of this work is a new approach for penetration testing targeting SQL Injection vulnerabilities. This choice is explained by the fact that among the vulnerabilities found in the 300 web services tested this is the most common and dangerous type, representing 84% of the vulnerabilities found. Furthermore, this is one of the most common types of vulnerabilities in database centric web applications [44][45][53]. The proposed approach consists of a set of tests based on malicious parameters (i.e., attacks) that is applied to disclose code vulnerabilities. Web services responses are analyzed based on well-defined rules that help confirming the existence of vulnerabilities and avoiding false positives. Comparing to existing web vulnerability scanners, our approach has three key improvements: 1) use a representative workload to exercise the services and understand the expected behaviour (i.e., the typical responses in the presence of valid inputs); 2) the set of attacks performed is a compilation of all

2

(15)

15 the attacks performed by a large set of scanners plus many attack methods that can be found in the literature; and 3) we apply well defined rules to analyze the web services responses in order to improve coverage and remove false positives. To demonstrate the effectiveness of our approach we implemented a prototype tool and used it to identify SQL Injection vulnerabilities in 262 publicly available web services and in one implementation of web services specified by the standard TPC-App performance benchmark [48]. Then, the results obtained were compared with the results obtained by web security scanners. Results shown that, for this particular type of vulnerability, our approach performs much better than commercial tools. 3

Although penetration testing is based on the effective execution of the code (i.e., provides a runtime view of the web service behaviour), vulnerabilities detection is based essentially on the analysis of the web services responses, which limits the visibility on the internal behaviour of the code. To overcome this problem we propose a new approach that combines the analysis of web services responses with the analysis of the web service behaviour during runtime for the detection of SQL and XPath injection vulnerabilities. In the development of this tool we decided to include XPath Injection as target vulnerability due to three key reasons: it is a high risk type of vulnerabilities; it is a very common vulnerability; and techniques to detect this type of vulnerability are similar to the ones to detect SQL Injection. Our approach is based in two main steps. First we generate and run a workload to exercise the web service and learn the profile of the SQL/XPath commands issued. Afterwards we apply a set of command injection attacks and observe the SQL/XPath commands being executed. This allows us to detect existing vulnerabilities by matching incoming commands during attacks with the valid set of commands previously learned. To demonstrate the effectiveness of our approach we implemented a prototype tool and used it to identify SQL and XPath Injection vulnerabilities in 9 services with multiple operations. Results show that our approach performs much better than existing penetration testing and static code analysis tools. In the experiments performed, it detected 100% of the known vulnerabilities (i.e., the vulnerabilities previously found by a team of security experts), while reporting no false positives. 4

1.1. Contributions

The goal of this dissertation work was to research existing and new mechanisms to evaluate web services’ security. The main contributions are:

 The study of the most common and dangerous types of vulnerabilities in web services environment. Many studies present the distribution of vulnerability types in web applications, but it was needed to study the differences existent in web services environment.

 The evaluation of existing vulnerability detection tools on web services environment, namely black-box testing tools as commercial web security scanners.

3

This w ork w as p resented in [6] and is d iscussed chap ters 4 and 5.

4

(16)

16

 A black-box testing tool to detect SQL Injection vulnerabilities. The proposed tool shown to be, in some cases, more effective than the present commercial security scanners by presenting a higher coverage while keeping the rate of false positives low. This tool can be very useful for both services’ providers and consumers as tests services from an external point of view.

 A gray-box testing tool to detect SQL Injection and XPath Injection vulnerabilities. The proposed tool shown to be able to outperform existing black-box and white-box vulnerability detection tools. This tool cannot be used by services’ consumers but is very useful for services’ providers, helping them to improve their services’ security before deploy them.

1.2. Thesis Structure

The document is divided in chapters as described following.

Chapter 2 presents some background and related work on the relevant areas to this work. The main topics are background on web services; software security testing with focus on web applications security testing; related existent vulnerability detection techniques and tools; related assessments on vulnerability detection tools effectiveness. Chapter 3 presents the work on assessing the effectiveness of commercial web security scanners in services environment. The evaluated scanners are considered as representative of the state of the art in black-box testing for web applications.

Chapter 4 presents a new black-box testing tool focused in SQL Injection vulnerabilities. The chapter includes a detailed description of the approach and its experimental evaluation.

Chapter 5 presents a new behaviour based gray-box testing tool focused in SQL and XPath Injection vulnerabilities. The chapter describes in detail the approach and its experimental evaluation.

Finally, chapter 6 concludes this thesis and briefly introduces the future work

(17)

17

Chapter 2 Background and Related Work

2.1. Web Services

A web service is a piece of business logic available on a network (usually Internet) and accessible through an Internet protocol such as HTTP or HTTPS by the exchange of XML-based messages according to Simple Object Access Protocol (SOAP) [13]. The service provides a set of operations and, in practice, each operation is a method with several input parameters. In each interaction the consumer (client) sends a request SOAP message to the provider (server). After processing the request, the server sends a response SOAP message to the client with the results.

A web service is described using a WSDL (Web Services Definition Language) [14] document. WSDL is a XML document that specifies the format of the messages to be exchanged between server and client. To facilitate the discovery of Web services, a discovery standard was developed. This standard, called Universal Description, Discovery and Integration (UDDI) allows applications to search for Web Services dynamically. When combined with WSDL, it is possible to discover and use new services at run-time without human intervention [42]. Web services’ providers only need to register web services and their description allowing potential users to discover it.

A web service is supported by a complex software infrastructure, which typically includes an application server, the operating system and external systems as data sources (DBMS, XML databases, etc) or even other web services. Figure 1 presents a simplified view of the typical structure of the web services system:

(18)

18

Web services can be implemented in a wide variety of architectures and technologies can interoperate with other technologies and software design approaches and can be adopted in an evolutionary manner without requiring major transformations to legacy applications [42]. This interoperability is one of the main causes of web services adoption. Service Oriented Architectures (SOA) is another cause to web services’ adoption. As with any new technology, this success comes with a level of increased risk [28].

“Web services are a technology that can be used to implement Service Oriented Architectures (SOA) and are increasingly becoming the SOA implementation of choice. For a SOA to truly meet its goals, applications developed must be secure and reliable.” [42].

2.2. Web Applications’ Threats And Security Vulnerabilities

Web services, as web applications in general, are so exposed that any existent security vulnerability will probably be uncovered and exploited becoming the entry point for malicious attacks. Traditional security mechanisms like network firewalls, intrusion detection systems (IDS), and use of encryption can protect the network but cannot mitigate attacks targeting web applications, even assuming that key infrastructure components such as web servers and database management systems (DBMS) are fully secure. For this reason, hackers are moving their focus from network to web applications where poor programming code represents a major risk. This can be confirmed by numerous vulnerability reports available in specialized sites like [40][31], etc.

Published studies show that, in general, web applications often present dangerous security flaws. In February 2007 Acunetix presented the results produced from the scanning of 3,200 websites during the previous year [2]. According to these results, 70% of the web sites scanned present high or medium risk vulnerabilities. Another interesting point is that in websites with high risk vulnerabilities the two most found vulnerabilities were SQL Injection (50% of the websites with high vulnerabilities) and Cross Site Scripting (42% of the websites with high vulnerabilities). The NTA's Annual Security Report 2008 [32] states that 25% of companies tested contain one or more high risk vulnerabilities. This number is lower than the 32% reported in 2007 [34]. Nevertheless, some sectors (finance, government, legal, retail and utilities) had increase the overall vulnerabilities found. In NTA's Annual Web Application Security Report 2008 [33], focused in web applications, states that 17% of the applications tested contain at least one high risk vulnerability and 78% of the applications tested contains medium risk vulnerabilities. These results cannot be generalized to web services environment, but show a high number of software being deployed without proper security cautions, including web applications.

About the vulnerabilities, the ten most critical web application security vulnerabilities are presented in 2007 [44] by the Open Web Application Security Project (OWASP Foundation) [35]. The two first vulnerabilities of this Top10 are the same two most found vulnerabilities by Acunetix’s study presented above. “A XSS flaw occurs whenever an application takes user supplied data and sends it to a web browser without first validating or encoding that content.” [44]. As many times web services works

(19)

19 without any attachment to web sites, XSS is not a priority in web services environment. On the other hand, Injection vulnerabilities are critical. An injection occurs when data from an input is used without any check as part of a command or a query. The most common injection flaw is SQL injection where user-supplied data can alter the structure of a SQL query that would be sent to a database. This way, a user can read or modify database data, accessing sensitive information. Similarly, XPath injection can compromise applications with the data stored in XML documents.

2.3. Security Testing

In order to minimize security issues, developers should apply best coding practices, perform security reviews of the code, execute penetration tests, use code vulnerability analyzers, etc. There are two main approaches to find web applications for vulnerabilities: white-box testing and black-box testing. Newer techniques generically named gray-box testing combine black-box and white-box testing to achieve better results. The following subsections detail these approaches and tools that implement each one of them.

2.3.1. White-box testing

White-box testing consists in the analysis of the source code of the web application without executing it. This can be done in one of two ways: manually in code reviews or by using an automated code analysis tools.

A code review is the process by which the programmers shows his code to his peers and they systematically examine it, searching for programming mistakes that can introduce bugs. A security code review is a code review that is specially targeted to find security vulnerabilities. Code reviews are the most effective way of making sure your code has a minimum number of bugs [36] and are a crucial procedure when developing software to critical systems. Nevertheless, they are usually very long, expensive and require that reviewers understand deeply about application security. That’s why new automatic review mechanisms appear, such as static code analysis tools [46].

The use of automated code analysis tools is often an easier and faster way to find bugs and vulnerabilities in the code. Static analysis tools vet software code, either in source or binary form, in an attempt to identify common implementation-level bugs. The analysis performed by existing tools varies depending on their sophistication, ranging from tools that consider only individual statements and declarations to others that consider the complete code. Among other usages (e.g., model checking and data flow analysis), these tools provide an automatic way for highlighting possible coding errors. The main problem of this approach is that exhaustive source code analysis may be difficult and cannot find all security flaws because of the complexity of the code.

The most common automated tools to perform white-box testing are usually named static code analyzers. The following paragraphs briefly introduce some of the most used and well-known static code analyzers. Some of them are commercial tools and other ones are free tools.

(20)

20

FindBugs [16] is “a program which uses static analysis to look for bugs in Java code” that is able to scan the bytecode of Java applications detecting, among other problems, security issues (including SQL injection). It is one of the most used tools for static code analysis.

Yasca (Yet Another Source Code Analyzer) [58] is “a framework for conducting source code analyses” in a wide range of programming languages, including java. Yasca includes two components. The first is a framework for conducting source code analyses and the second is an implementation of that framework that allows integration with other static code analyzers (e.g., FindBugs, PMD, and Jlint).

IntelliJ IDEA [24] is a commercial tool that provides a powerful IDE for Java development and includes “inspection gadgets” plug-ins with automated code inspection functionalities. IntelliJ IDEA is able to detect security issues, such as SQL problems, in java source code.

Fortify 360 is commercialized by Fortify Software [18] and includes a Source Code Analyzer (SCA) that “provides root-cause identification of vulnerabilities in source code SCA is guided by the largest and most comprehensive set of secure coding rules and supports a wide array of languages, platforms, build environments and integrated development environments (IDEs).” [18].

Pixy [38] is a free and open source program that performs automatic static code analysis of PHP 4 source code, aimed at the detection of XSS and SQL injection vulnerabilities. As referred in Pixy’s webpage, “Pixy takes a PHP program as input, and creates a report that lists possible vulnerable points in the program, together with additional information for understanding the vulnerability”.

2.3.2. Black-box testing

The black-box testing approach consists in the analysis of the execution of the application in search for vulnerabilities. In this approach, also known as penetration testing, the tester does not know the internals of the web application and it uses fuzzing techniques over the web HTTP requests. This approach is more effective than white-box testing on larger units of code. The tester needs no knowledge of implementation and tests the inputs of application from user’s point of view. The number of tests needed to can reach hundreds or even thousands for each vulnerability type. Once this rarely is possible to be done manually, automated test tools are used to perform. Penetration testing tools provide an automatic way to search for vulnerabilities avoiding the repetitive and tedious task of doing hundreds or even thousands of tests by hand for each vulnerability type.

Despite the automated tools, still not being possible to test all possible input streams, as it would take too much time. So, as soon as software specifications are complete, test cases can be designed to have the biggest coverage and representativeness possible. This test approach may leave program paths untested and can lead to unnecessary repetition of tests between developers and testers.

(21)

21 The most common automated security testing tools used in web applications are generally referred to as web security scanners (or web vulnerability scanners). Web security scanners are often regarded as an easy way to test applications against vulnerabilities. In fact, security scanners provide an automatic way to search for vulnerabilities avoiding the repetitive and tedious task of doing hundreds or even thousands of tests by hand for each vulnerability type. These scanners have a predefined set of tests that they adapt to applications to be tested, saving user from define all the tests to be done. User only needs to configure the scanner and let him test the application. Once the test is completed, scanner reports existing vulnerabilities if any detected.

Due to evolution of web services adoption, security scanners started to include support for web services security scanning. Most of these scanners are commercial tools, but there are also some free application scanners often with limited use, since they lack most of the functionalities of their commercial counterparts.

The most popular free security scanners that support web services security assessment are Foundstone WSDigger [19] and WSFuzzer [55]. WSDigger is a free open source tool developed by Foundstone that executes automated penetration testing in web services. Only one version of this software was released and date of December 2005. This version contains sample attack plug-ins for SQL Injection, cross site scripting (XSS) and XPath Injection. This tool was released as open-source to encourage users to develop and share their own plug-ins and is simple to edit test files to add new test cases. WSFuzzer is a free open source program that currently has HTTP based SOAP services as the main target. This tool was created based on some real-world manual SOAP penetration testing work, automating it. Nevertheless, it does not meant to replace a solid manual human analysis. One problem of this tool is that its configuration is very complex.

The main problem common to these open source tools is the fact that they do not detect vulnerabilities, only execute automated tests. In other words, tools just execute attacks to web service and logs the responses leaving to user the task of examine these logs and determine if application is vulnerable. It requires the user to be an “expert” in security field and to spend a huge amount of time to examine all the results. This is the cause for these tools not being tested in our research work.

In commercial scanners field three brands lead the market: HP WebInspect [21], IBM Rational AppScan [23] and Acunetix Web Vulnerability Scanner [3]. Next paragraphs introduce these scanners and their most important properties.

HP WebInspect “performs web application security testing and assessment for today's complex web applications, built on emerging Web 2.0 technologies. HP WebInspect delivers fast scanning capabilities, broad security assessment coverage and accurate web application security scanning results” [21]. This tool includes pioneering assessment technology, including simultaneous crawl and audit (SCA) and concurrent application scanning. It is a broad application that can be applied for penetration testing in web-based applications. HP WebInspect supports web services scanning since 2000 (WebInspect classic v1.0).

(22)

22

IBM Rational AppScan “is a leading suite of automated Web application security and compliance assessment tools that scan for common application vulnerabilities” [23]. This tool is suitable for users ranging from non-security experts to advanced users that can develop extensions for customized scanning environments. IBM Rational AppScan can be used for penetration testing in web applications, including web services. When used to scan web services, this application has the particularity of requiring that the user executes a valid invocation of each operation to be tested. The scanner stores the response and compares it with responses to the tests. In one hand, this is an added value, because it will improve the knowledge of the scanner when evaluating service responses. On the other hand, it can be a delaying task for the user, because it can consume a considerable amount of time, and sometimes the user doesn’t have the knowledge required to execute this operation, once he ignores the internals of the web service. IBM Rational AppScan supports web services scanning since July 2006 (v6.5). Acunetix Web Vulnerability Scanner (WVS) “is an automated web application security testing tool that audits your web applications by checking for exploitable hacking vulnerabilities” [3]. Acunetix WVS can be used for penetration testing in web applications or web services, is simple to use and to configure and provides a command line interface, also very simple to use. Despite this is not any kind of “commercial review” still being an interesting point of this application the fact that its price is way lower than its counterparts’ prices. Acunetix WVS was first released in July 2005 and supports web services scanning since June 2007 (v5.0.70604).

Another related box technique is WAVES, proposed on [22]. WAVES is a black-box technique for testing web applications for SQL-Injection vulnerabilities. The technique is based in a reverse engineering process that identifies data entry points of a Web application and attacks them with malicious patterns. An algorithm is proposed to allow “deep injection” and to eliminate false negatives. The problem is that the technique can only be applied to web applications (not to web services) and ignores the user knowledge about the application being tested.

2.3.3. Gray-box testing

Other techniques combine black-box testing with white-box testing to overcome the limitations of each other achieving better results. Of course, techniques based on this approach can only be used by the application owner (or the web service provider, in web services context). This combination can be done in very different ways.

Simple techniques use some preliminary analysis of the code to find possible vulnerable points or to establish the attacks to perform in a later phase. This way the attacks can be manipulated in order to exercise the parts of code detected as error-prone in the analysis phase.

Other possibility, more elaborated, is to perform source code or bytecode instrumentation in a way that can be obtained more information about the reaction of the application to the tests. This technique is used in “Acunetix AcuSensor Technology” [1]. This technology combines black-box scanning with feedback obtained during the test execution. This feedback is provided by sensors previous placed inside the source code or bytecode. Acunetix states that by using this technique is possible to find more

(23)

23 vulnerabilities, to indicate in the code exactly where they are and to report less false positives. This technology is only available to web application, specifically .NET and PHP web applications. In case of .NET this technology can be injected in bytecode.

2.4. Assessing Vulnerability Detection Tools

In what concerns to web security scanners, previous research suggests that their effectiveness in the detection of vulnerabilities varies a lot, being often unsatisfactory. In [17] it is proposed a method to evaluate and benchmark automatic web vulnerability scanners in web application’s environment using software fault injection techniques. Software faults are injected in the application code and the web vulnerability scanning tool under evaluation is executed, showing their strengths and weaknesses, concerning coverage of vulnerability detection and false positives. The study focused in SQL Injection and Cross Site Scripting (XSS) types of vulnerabilities. Three leading commercial scanning tools were evaluated and the results showed that in general the coverage is low and the percentage of false positives is very high (ranging from 20% to 77%). However, this study was focused on a specific family of applications, namely database centric web based applications written in PHP, and the results obtained cannot be easily generalized, especially if we take into account the specificities of web services environments.

In [46] is proposed a preliminary version of a benchmark to static analysis tools effectiveness. This is based on a study realized over code analysis tools freely available on internet. This benchmark is based on an application developed by the authors containing representative vulnerabilities. The tools are run over the application to find the existent vulnerabilities. The experience shown that each tool find only few classes of vulnerabilities. Using these results a new tool was developed to aggregate the results of the benchmarked tools originating a new tool with higher coverage and less false positives.

In [52] authors evaluated three bug finding tools and compared their effectiveness with a review team inspection. The tools achieved higher efficiency than the review team in detecting software bugs (the study did not consider security issues) in five industrial Java-based applications, but all the tools presented false positive rates higher than 30%.

(24)

(25)

25

Chapter 3 Using Web Security Scanners to Detect

Vulnerabilities in Web Services

Commercial web security scanners are widely used in development and are considered as representative of the state of the art in black-box testing for web applications [3]. These scanners are regarded as an easy way to test applications searching for security vulnerabilities. However, previous research suggests that the effectiveness of scanners in the detection of vulnerabilities varies a lot, sometimes with unsatisfactory results [17]. To confirm this in the specificities of web services’ environment, we have used four well know vulnerability scanners to identify security flaws in 300 public web services. The goal was to study the effectiveness of the scanners and to try to identify common types of vulnerabilities in web services environments. A large number of vulnerabilities have been observed, which confirms that many services are deployed without proper security testing. Additionally, the differences in the vulnerabilities detected and the high number of false-positives, and low coverage observed highlight the limitations of web vulnerability scanners on detecting security vulnerabilities in web services.

In summary, the experiments were conducted to answer to the following three questions:

 What is the coverage of the vulnerability scanners tested when used in a web services environment?

 What is the false-positive rate of the web vulnerability scanners tested when used in a web services environment?

 What are the most common types of vulnerabilities in web services environments?

The following sections detail the experimental study preformed and present and discuss the results obtained.

3.1. The Experimental Study

Our experimental study consisted of four steps:

1) Preparation: select the security scanners to use and publicly available web services to scan.

2) Execution: use the security scanners to scan the services to identify potential vulnerabilities.

(26)

26

3) Verification: perform manual testing to confirm that the vulnerabilities identified by the scanners do exist (i.e., are not false positives).

4) Analysis: analyze the results obtained and systematize the lessons learned. First of all, the selected security scanners are four commercial web vulnerability scanners widely used, including two different versions of a specific brand. The three commercial brands had already been presented in chapter 2 and are:

 HP WebInspect [21];

 IBM Rational AppScan [23];

 Acunetix Web Vulnerability Scanner [3];

For the results presentation we have decided not to mention the brand and the versions of these scanners to assure neutrality and because commercial licenses do not allow in general the publication of tool evaluation results. This way, from this point to the rest of the document, scanners presented above will be referred to as VS1.1, VS1.2, VS2, and VS3 (without any order in particular). Vulnerability scanners VS1.1 and VS1.2 refer to the two versions of the same product. Using the free scanners was also considered, but to that be possible would be necessary to examine results by hand (as explained above) or develop some automatic ways to do that. As both solutions needed time, this option was discarded due to time constraints.

Second, the selection of web services to be tested. The established goal was to test 300 web services, randomly selected. The random selection of the services is due to two main reasons: allow a fair comparison between the scanners and allow us to gather information about vulnerability distribution in web services context. In other words, if the services were selected based on knowledge about their vulnerabilities, the vulnerability distribution would not be representative of their real status in web services environment. Moreover, if that knowledge was obtained using one of the tools under evaluation, that tool would be benefited in the evaluation results. The process to select the set of services to be tested is described in detail in the next paragraphs.

The first step was to identify a large set of web services. The first source was a well known web site that lists publicly available web services, xmethods.net [57]. From this list we have selected the first set of 450 web services. Then we used the seekda [41] to discover additional services. seekda is a portal that allow users to search for public web services based on the services’ details (description, provider, country, etc). To discover services we have used a large set of generic keywords, ranging from popular tags (e.g., business, tourism, commercial, university, etc) to queries with countries (e.g., country:AR, country:PT, country:US, etc) and keywords related to company names (e.g., Oracle, Sun, Microsoft, Google, Acunetix).

The resulting list included 6180 web services. From this universe, we have to select 300 web services to test. For this we attributed an ID to each service and generated 1000 random numbers between 0 and 6179. The first 300 numbers of this list are the IDs of

(27)

27 services to be tested. When some service has to be discarded we pick the next number on random numbers list. To achieve our goal, 316 services had been discarded distributed by the following reasons:

 Invalid/malformed WSDL: at least one of the scanners could not parse the WSDL description. This can show that a high number of services disrespect standards, but can also show that the scanners are not very tolerant to WSDL ambiguities. Happened in 117 cases.

 Unable to retrieve WSDL: at least one of the scanners could not find the WSDL. Happened in 30 cases.

 No methods found: at least one of the scanners aborts the scanning because the web service has no operations to scan. Happened in 20 cases.

 Authentication required: some operations require authentication, which means that the service cannot be fully tested. This prevents the observation of vulnerabilities in potentially interesting functionalities of services (the private functionalities), which would obviously be of high interest. Occur 40 times.

 Unhandled exception: the tests abort and/or one of the scanners closes unexpectedly and repeatedly in the same phase of the scanning. Occur 19 times.

 Communication errors: at least one of the scanners cannot complete requests due to problems in HTTP Request or SSL Connection Error. Occur 50 times.

 Scanning problems: at least one of the scanners reported a generic error like “unable to add transaction” or “unable to render object”. Happened in 24 cases.

 Testing duration: due to practical reasons, we decided to drop the services for which the scanning process takes more than two hours. Happened in 15 cases. After this process, a final set of 300 publicly available services was obtained. The following bullets point out some important aspects concerning the web services selected:

 Several technologies were considered, including .NET, Java, and php.

 Web services are owned by different relevant parties, including Microsoft, Google, and Xara.

 Some web services implement the same functionality (e.g., Text Disguise and Free Captcha Service).

 A small number of web services are used in real businesses in the field (e.g., Portuguese Postal Office Orders Cost, Mexican Postal Codes).

(28)

28

After the selection of scanners and services, the test phase began. Tests were executed with scanners configured to their complete web services profiles. The initial list of web services and list of tested services were not included in this document but they are available at [8] to be consulted. The following sections present and discuss the results obtained and the lessons learned during this study.

3.2. Overall Results

Table 1 presents the overall results of the study. For each scanner it is presented the total number of vulnerabilities and the number of services in which those vulnerabilities were found. The scanners pointed six different types of vulnerabilities, namely:

 SQL Injection: it is possible “to alter the construction of backend SQL statements” [53].An attacker can read or modify database data and, in some cases, execute database administration operations or commands in system [45].

 XPath Injection: it is possible to modify an XPath query to “be parsed in a way differing from the programmer's intention” [53]. Attackers may gain access to information in XML documents [45].

 Code Execution: it is possible to manipulate the application inputs to trigger server-side code execution [45]. An attacker can exploit this vulnerability to execute malicious code in the server machine.

 Buffer Overflow: it is possible to manipulate inputs in such a way that causes buffer allocation problems, including overwriting of parts of the memory [45]. An attacker can exploit this causing Denial of Service or, in worst cases, “alter application flow and force unintended actions” [53].

 Username/Password Disclosure: the web service response contains information related to usernames and/or passwords. An attacker can use this information to get access to private data [45].

 Server Path Disclosure: the response contains a fully qualified path name to the root of the server storage system. An attacker can use this info to discover the server file system structure and devise other security attacks [45].

Vulnerability Types VS1.1 VS1.2 VS2 VS3 # Vuln. # WS # Vuln. # WS # Vuln. # WS # Vuln. # WS

SQL Injection 217 38 225 38 25 5 35 11

XPath Injection 10 1 10 1 0 0 0 0

Code Execution 1 1 1 1 0 0 0 0

Possible Parameter Based Buffer Overflow 0 0 0 0 0 0 4 3 Possible Username or Password Disclosure 0 0 0 0 0 0 47 3

Possible Server Path Disclosure 0 0 0 0 0 0 17 5

Total 228 40 236 40 25 5 103 22

(29)

29 An additional result provided by the scanners is related to application errors obtained due to invalid parameters. Scanners classify these situations as low importance security issues and point that if these errors or exceptions are present in service’s documentation, they are probably false positives and should be ignored. In fact, most part of times, these errors are due to the fact that input parameters are not correctly checked (i.e., the service may have robustness problems). As shown in Figure 2, the four scanners detected different application errors (although the two versions of the same scanner have similar results). This is obviously dependent on the tests performed by each scanner. Although in few cases application errors can be maliciously explored (e.g., if the error message discloses sensitive information about the service code or structure), in most cases they are not related to security vulnerabilities. Thus, in the present work we have decided to exclude them from further analysis.

Figure 2 - Application errors detected.

As we can see in Table 1, different scanners report different types of vulnerabilities. The only exception is SQL Injection that is reported by all scanners. This is a first indicator that scanners implement different forms of penetration tests and that the results from different scanners may be difficult to compare. Nevertheless, results suggest that in order to have a good coverage the user will have to use several scanners instead of relying in only one.

Scanners VS1.1 and VS1.2 (two different versions of the same brand) are the only ones that detected XPath Injection vulnerabilities. An important aspect is that, when compared to SQL Injection, the number of XPath-related vulnerabilities is quite small. In fact, XPath vulnerabilities were detected in a single service, indicating that most web services make use of a database instead of XML documents to store information.

Scanners VS1.1 and VS1.2 detected a code execution vulnerability. This is a particularly critical vulnerability that allows attackers to execute code in the server. After discovering this vulnerability we performed some manual tests and we were amazed by the possibility of executing commands (e.g., ‘cat /etc/passwd’, ‘ls -la’) and get the corresponding answer in a readable format.

2931 2913 1090 ₁₀₀₂ 0 500 1000 1500 2000 2500 3000 3500 VS1.1 VS1.2 VS2 VS3

(30)

30

VS3 was the only one pointing vulnerabilities related to buffer overflow, username and password disclosure, and server path disclosure.

As mentioned before, SQL Injection is the only type of vulnerability that was detected by the four scanners used. However, different scanners reported different vulnerabilities in different web services. In fact, the number of SQL Injection vulnerabilities detected by VS1.1 and VS1.2 is much higher than the number of vulnerabilities detected by VS2 and VS3. Thus we decided to look at this vulnerability type in more detail.

The first analyze to SQL Injection is represented in Figure 3, where can be seen the intersections between vulnerabilities found by different scanners. The intersection areas of the circles in Figure 3 represent the number of vulnerabilities detected by more than one scanner (the number of vulnerabilities detected is shown; zero is the value when no number is presented). Note that the area of each circle is roughly proportional to the number of vulnerabilities detected, but there is no correspondence between the size of the intersection areas and the number of vulnerabilities (it is too complex to represent graphically).

Figure 3 clearly shows that the four scanners detected different sets of SQL Injection vulnerabilities and the differences are considerable, pointing again to relatively low coverage of each vulnerability scanner individually. In fact, even for VS1.1 and VS1.2, two consecutive versions of the same scanner, there are considerable differences. VS1.1, the older version, detected 19 SQL Injection vulnerabilities that were not detected by VS1.2. On the other hand, VS1.2 detected 27 vulnerabilities that were not detected by VS1.1.

3.3. False Positive Analysis

The results presented so far do not consider false positives (i.e., situations where scanners detected a vulnerability that in the reality does not exist). However, it is well known that false positives are very difficult to avoid. This way, we decided manually confirm the existence (or not) of each vulnerability detected.

(31)

31 Confirming the existence of a vulnerability without having access to the source code is a difficult task. Thus, we defined a set of rules and corresponding checks to classify the vulnerabilities detected by the scanners in three groups: a) Confirmed false positives, b) Confirmed vulnerabilities, and c) Doubtful.

Detected vulnerabilities are classified as false positives if meet one of the following cases:

1) For SQL Injection vulnerabilities, if the error/answer obtained is related to an application robustness problem and not to a SQL command (e.g., a NumberFormatException).

2) The error/value in the web service response is not caused by the elements "injected" by the scanner. In other words, the same problem occurs when the service is executed with valid inputs.

3) For path and username/password disclosure, the information returned by the service is equal to the information submitted by the client (e.g., the vulnerability scanner) when invoking the web service. In other words, there is no information disclosure.

Detected vulnerabilities are classified as confirmed vulnerabilities if satisfy one of the next conditions:

1) For SQL Injection vulnerabilities, if it is possible to observe that the SQL command executed was invalidated by the values "injected" by the scanner (or manually). This is possible if the SQL command or part of it is included in the web service response (e.g., stack trace).

2) For SQL Injection vulnerabilities, if the “injected” values lead to exceptions raised by the database server.

3) If it is possible to access unauthorized services or web pages (e.g., by breaking the authentication process using SQL Injection).

4) For Path disclosure, if it is possible to observe the location of folders and files in the server.

5) For XPath Injection, if the “injected” values lead to exceptions raised by the XPath parser.

6) For Buffer Overflow, if the server does not answer to the request or raises an exception specifically related to buffer overflow.

If none of these rules can be applied then there is no way to confirm whether a vulnerability really exists or not. These cases were classified as doubtful. Figure 4 shows the absolute results for SQL Injection vulnerabilities. Table 2 presents false positives and doubtful vulnerabilities in relative terms. As we can see, the percentage of vulnerabilities that we were not able to confirm (doubtful cases) is low for VS1.1,

(32)

32

VS1.2, and VS3 (always less than 15%), but considerably high for VS2 (32%). This means that the false positive results are relatively accurate for the first three scanners, but it is an optimistic figure (zero false positives) for scanner VS2. Obviously, we can also read the false positive results shown in Figure 4 as a range, going from an optimistic value (confirmed false positives) to a pessimistic value (confirmed false positives + doubtful cases).

Figure 4 - False Positives and Doubtful vulnerabilities.

Scanner % False Positives % Doubtful

VS1.1 40,09% 6,45%

VS1.2 36,89% 11,56%

VS2 0% 32%

VS3 25,71% 14,29%

Table 2 - False Positives and Doubtful vulnerability percentages.

The number of (confirmed) false-positives is high for scanners VS1.1 and VS1.2, and is also high for VS3, in relative terms. Scanner VS2 shows zero confirmed false positives, but it detected a fair percentage (8 out of 25) of vulnerabilities that were classified as doubtful, thus a pessimistic interpretation of results is that 8 out of 25 vulnerabilities may be false positives. Obviously, the low number of vulnerabilities detected by VS2 and VS3 (25 and 35 respectively) also limits the absolute number of false positives. Table 3 presents the false positive results for the other vulnerabilities. In this case, we were able to confirm the existence (or inexistence) of all vulnerabilities and no doubts remained. An interesting aspect is that all XPath injection and Code Execution vulnerabilities were confirmed. On the other hand, all vulnerabilities related to username and password disclosure were in fact false positives (in all cases the username/information information returned is equal to the one sent by the scanner).

116 116 17 21 14 26 8 5 87 83 9 0 25 50 75 100 125 150 175 200 225 VS1.1 VS1.2 VS2 VS3

Confirmed False Positives Doubtful

(33)

33

Vulnerability Scanner Confirmed False Positives

XPath Injection VS1.1 & VS1.2 10 0

Code Execution VS1.1 & VS1.2 1 0

Possible Parameter Based Buffer Overflow VS3 1 3

Possible Username or Password Disclosure VS3 0 47

Possible Server Path Disclosure VS3 16 1

Table 3 - False positives for other vulnerability types.

Figure 5 presents the SQL Injection vulnerabilities intersections after removing the false positives. The doubtful situations were in this case considered as existing vulnerabilities (i.e., optimistic assumption from the point of view of scanners detection effectiveness). Results clearly show that, even if we manually remove the false positives, the four scanners still report different vulnerabilities. An interesting result is that three vulnerabilities were detected by VS1.1 and were not detected VS1.2 (the newer version of the scanner). The reverse also happens for 15 vulnerabilities, which is expectable as a newer version is expected to detected more vulnerabilities than an older one (but that should happen without missing any of the vulnerabilities identified by the older version, which was not the case). These results called our attention and we tried to identify the reasons. After analyzing the detailed results we concluded that all of these 18 vulnerabilities are in the group of the doubtful ones (maybe they are really false positives, but we were not able to demonstrate that), preventing us from drawing a definitive conclusion.

Figure 5 - SQL Injection vulnerabilities without False Positives.

3.4. Coverage Analysis

A key aspect is to understand the coverage of the vulnerabilities detected. Coverage compares the number of vulnerabilities detected against the total number of vulnerabilities. Obviously, in our case it is impossible to know how many vulnerabilities were not disclosed by any of the scanners (we do not have access to the

(34)

34

source code). Thus, it is not possible to calculate the coverage. However, it is still possible to make a relative comparison based on the data available.

In practice, we know the total number of vulnerabilities detected (which correspond to the union of the vulnerabilities detected by the four scanners after removing the false positives) and the number of vulnerabilities detected by each individual scanner. Based on this information it is possible to get an optimistic coverage indicator for each scanner (i.e., the real coverage will be lower than the value presented). Obviously, this is relevant only for SQL Injection vulnerabilities as it is the only type that is detected by all the scanners.

Table 4 presents the coverage results. As shown, 149 different SQL Injection vulnerabilities were detected (as before, we decided to include the doubtful situations as existing vulnerabilities). Each scanner detected a subgroup of these vulnerabilities, resulting in partial detection coverage. VS1.1 and VS1.2 presented quite good results. On the other hand, the coverage of VS2 and VS3 is very low.

Scanner # SQL Injection Vulnerabilities Coverage %

VS1.1 130 87.2%

VS1.2 142 95.3%

VS2 25 16.8%

VS3 26 17.4%

Total 149 100.0%

Table 4 - Coverage for SQL Injection.

3.5. Lessons Learned

The results presented before allowed us to observe some interesting aspects. The first observation is that different scanners detected different types of faults. SQL Injection was the only type that was detected by all scanners. The two scanners of the same brand (VS1.1 and VS1.2) were the only ones that detected XPath and code execution vulnerabilities. Only one scanner (VS3) detected vulnerabilities related to buffer overflow, username and password disclosure, and server path disclosure. VS2 only detected SQL Injection vulnerabilities.

SQL Injection vulnerabilities are the dominant type in the web services tested (see Figure 6). However, different scanners detected different vulnerabilities of this type. In fact, VS1.1 and VS1.2 detected a huge number of vulnerabilities (215 and 225 respectively) while VS2 and VS3 detected a very low number (25 and 35 respectively). A key observation was a very large number of false positives. In fact, for three of the scanners the percentage of false positives was more than 25%. VS2 presented zero false positives, but 8 out of the 25 SQL Injection vulnerabilities detected by this scanner remained as doubtful (i.e., could not be manually confirmed as real vulnerabilities nor as false positives). This reduces the confidence on the precision of the vulnerabilities detected

(35)

35 A very low coverage, lower than 18%, was observed for two of the scanners (VS2 and VS3), while the other two scanners (VS1.1 and VS1.2) present a coverage superior to 87%. Note that this value represents an optimistic coverage, as the real coverage of the tested scanners (at least for the 300 web services used in the experiments) is definitely lower than the value observed, with many vulnerabilities probably remain undetected. Figure 6 presents the final distribution of vulnerabilities per type, after removing the confirmed false positives but including the doubtful cases (i.e., optimistic evaluation of the scanners). As the doubtful cases only affect the SQL Injection, it means that the number of SQL injection vulnerabilities could be overestimated. Scanners have found 177 different vulnerabilities in 25 different services, which represent approximately 8.33% of the tested services. As mentioned before, the predominant vulnerability is SQL Injection, representing 84.18% of the vulnerabilities found. This is a very important observation due to the high number of cases found and the high severity of this vulnerability.

(36)

(37)

37

Chapter 4 A Penetration Testing approach to Detect SQL

Injection Vulnerabilities in Web Services

The results presented in chapter 3 show that web security scanners are not an effective way to perform a web services security evaluation and suggested an open field to develop tools with an effective performance on the detection of security flaws in web services. This way, in this chapter we propose a new approach for penetration testing targeting SQL Injection vulnerabilities. The experimental evaluation of the approach demonstrates that it is possible to develop a vulnerability scanner for web services that performs much better than the commercial ones currently available.

The proposed approach for SQL Injection vulnerabilities detection follows the black-box testing model. This way, it tests web services from an external point of view. This fact is particularly interesting in web services’ environment because it allows to be used both by web services’ providers and consumers, benefiting both: the provider can solve security vulnerabilities in its own services and the consumer can assess the services that are planning to use.

In second place, the approach is focused in SQL Injection vulnerabilities because among the high risk vulnerabilities found during the experiments this is the most common and dangerous type, representing 84% of the vulnerabilities found. Furthermore, this is one of the most common types of vulnerabilities in database centric web applications [44] [45] [53].

The proposed approach consists of a set of tests based on malicious parameters (i.e., attacks) that is applied to disclose code vulnerabilities. Web services responses are analyzed based on well-defined rules that help confirming the existence of vulnerabilities and avoiding false positives. Comparing to existing web vulnerability scanners, our approach has three key improvements:

1) Uses a representative workload to exercise the services and understand the expected behaviour (i.e., the typical responses in the presence of valid inputs); 2) The set of attacks performed is a compilation of all the attacks performed by a

large set of scanners plus many attack methods that can be found in the literature;

3) Applies well defined rules to analyze the web services responses in order to improve coverage and remove false positives;

This approach to detect SQL Injection vulnerabilities in web services code is based on a tool that implements the following set of generic steps (described in detail in the following subsections):