Security Testing

Testing can be classified utilizing the three dimensions of flow, scales, and characteristics [143] shown in Figure 10. Test flow explains where tests are derived from, scripting/coding (white- box) or requirements (black-box). Test scale describes the granularity of the system under test (SUT) and can be unit testing or anything up to system testing. The test characteristics are the reason or purpose for designing and executing a test, for example testing the system’s functionality, robustness, capacity, or usability.

Figure 10: Types of testing in SE [143].

Model-based testing is “the automation of test design”. Tests are generated automatically from a model of SUT [144]. Because test suites are derived from models and not from source code, model-based testing is usually seen as a form of black-box testing [144].

Researchers in SE use static and dynamic analysis approaches to locate vulnerable code and analyze known security vulnerabilities (e.g., XSS and CSRF). A common use of SVDBs in this context is using application vulnerabilities for validation and to gain security knowledge.

Identifying vulnerabilities and ensuring security functionality by security testing is a widely applied measure to evaluate and improve the security of software. Software security testing is a process intended to reveal flaws in the security mechanisms of information systems that protect data and maintain functionality as intended [145].

Our survey provides an overview of the security testing techniques used in SE research. Table 8 summarizes the articles classified in this SE task, showing the title of each article, publication year, name of the used SVDB, and what the SVDB was used for.

Table 8: SE articles using SVDBs in security testing task

Reference Title of Paper Year SVDBs How SVDBs Used

Stivalet et al.

[78] Large Scale Generation of Complex and Faulty PHP Test

Cases

2016 CWE and

OWASP Used vulnerabilities information for comprehending security

attacks. Ceccato et al.

[76] SOFIA: An Automated Security Oracle for Black-Box Testing of

SQL-Injection Vulnerabilities

2016 CVE Downloaded vulnerable web

applications and web services affected by SQL injection. Pham et al.

[77] Model-Based Whitebox Fuzzing for Program Binaries 2016 OSVDB and CVE Downloaded vulnerable projects.

Palsetia et al.

[53] Securing Native XML Database-Driven Web Applications from

XQuery Injection Vulnerabilities

2016 OWASP Used security guidelines.

Bozic et al.

[54] Evaluation of the IPO-Family Algorithms for Test Case

Generation in Web Security Testing

2015 OWASP and

Exploit-DB Downloaded vulnerable web applications.

Appelt et al.

[146] Behind an Application Firewall, Are We Safe from SQL Injection

Attacks?

2015 OWASP Used the database as a knowledge

source to comprehend SQL injection attack.

Pham et al.

[147] Hercules: Reproducing Crashes in Real-World Application Binaries 2015 CVE Downloaded selected CVEs vulnerabilities reports.

Aydin et al.

[148] Automated Test Generation from Vulnerability Signatures 2014 OWASP Downloaded vulnerable web applications.

Hossen et al.

[149] Automatic Generation of Test Drivers for Model Inference of

Web Applications

2013 OWASP Downloaded vulnerable web

applications. Blome et al.

[150] VERA: A Flexible Model-Based Vulnerability Testing Tool 2013 OWASP Downloaded vulnerable web applications.

Lebeau et al.

[151] Model-Based Vulnerability Testing for Web Applications 2013 CAPEC and OWASP Used vulnerability knowledge and types of security attacks.

Buchler et al.

[152] SPaCiTE -- Web Application Testing Engine 2012 OWASP Downloaded Webgoat, a vulnerable web application.

Zhang et al.

[153] SimFuzz: Test Case Similarity Directed Deep Fuzzing 2012 NVD, SecurityFocus,

and

SecurityTracker

Extracted 100 buffer overflow vulnerabilities.

Shahriar et al.

[154] MUTEC: Mutation-Based Testing of Cross Site Scripting 2009 OSVDB Downloaded five open source web applications suffering from Cross

The surveyed papers in this SE task used SVDBs information to perform security testing and techniques in different aspects such as black-box testing (including model-based testing) as discussed in Ceccato et al. [76], Palsetia et al. [53], Appelt et al. [146], Aydin et al. [148], Hossen et al. [149], Blome et al. [150], Lebeau et al. [151], and Buchler et al. [152].

Black box and model-based testing. Black-box and model-based security testing are testing

techniques that describe how a system securely behaves in response to an action (determined by a model). Ceccato et al. [76] introduced SOFIA, a security oracle for black-box testing of SQL- injection vulnerabilities. The main purpose of SOFIA is to detect types of SQLi attacks. The proposed approach validated vulnerable web applications that use SQL relational database and has known vulnerabilities published in the CVE dataset.

Palsetia et al. [53] proposed a black-box fuzzing approach to detect different types of XQuery injection vulnerabilities in web applications driven by “native XML databases”35_{. The}

primary objective of the proposed method for detecting XQuery injection vulnerabilities is based on OWASP guidelines in native XML database-driven web applications.

Appelt et al. [146] focused on web application firewalls and SQL injection attacks. They presented a black-box (machine learning-based) testing approach to detect holes in firewalls that let SQL injection attacks bypass. They developed a tool that implements the approach and evaluated it on ModSecurity36_{, a widely used application firewall provided by the OWASP}

project.

Aydin et al. [148] showed that vulnerability signatures computed for deliberately insecure web applications (developed for demonstrating different types of vulnerabilities) can be used to generate test cases for other applications. Their proposed approach is a black-box specification- based testing approach. The authors used a deliberately insecure web application called Damn Vulnerable Web Application (DVWA) listed in the OWASP broken web applications project.

Hossen et al. [149] proposed automatic generation of test drivers for model inference of web applications. The authors have applied the method on WebGoat. WebGoat is organized in lessons; the goal of a lesson is to show a type of vulnerability and its corresponding attack. The

35 _{http://www.rpbourret.com/xml/ProdsNative.htm}

authors chose the stored cross-site scripting (XSS) lesson, which has a demonstrated vulnerability in the Top10 within OWASP.

Blome et al. [150] introduced VERA—a flexible model-based vulnerability testing tool. The proposed method is a tool that allows users to define attacker models where the payloads and the behavior are separated and that abstract away from low-level implementation details such as HTTP requests. The researchers give two examples of Injection flaws: Cross Site Scripting and SQL Injection using information from WebGoat extracted from the OWASP database. Lebeau et al. [151] introduced model-based vulnerability testing for web applications. The approach is based on a mixed modeling of the application under test; the specification indeed captures some behavioral aspects of the web application and includes vulnerability test purposes to drive the test generation algorithm. This approach is illustrated with the widely-used DVWA example.

Buchler et al. [152] presented a model checking tool called SPaCiTE - Web Application Testing Engine. The proposed tool relies on a dedicated model-checker for security analyses that generates potential attacks with regard to common vulnerabilities in web applications. The authors applied SPaCiTE to Role-Based-Access-Control (RBAC) and Cross-Site Scripting (XSS) lessons of WebGoat, an insecure web application maintained by OWASP.

Hybrid approaches. We found papers that used a combination of different testing approaches,

such as model-based white box testing approach (e.g., Pham et al. [77]), and an article that combined different approaches such as black box fuzzing, code analysis, and combination (i.e., combinatorial) testing (e.g., Zhang et al. [153]).

Pham et al. [77] presented Model-based Whitebox Fuzzing (MoWF), an automated testing technique for program binaries that process structured inputs. MoWF is a combination of model- based black box fuzzing and white box fuzzing. They evaluated their approach on 13 vulnerabilities in 8 program binaries with 6 separate file formats, and compared the explored vulnerabilities with ones from OSVD and CVE public databases.

Zhang et al. [153] proposed a fuzzing approach aimed to produce test inputs to explore deep program semantics. The fuzzing process integrates techniques from black-box fuzzing, code analysis, and combination testing. The main purpose of the approach is to detect memory corruption vulnerabilities such as buffer overflow and pointer out-of-boundary operations. The

authors used top 100 buffer overflow vulnerabilities from searching the vulnerabilities of NVD, SecurityFocus, and SecurityTracker.

Others. Other papers used SVDBs information in testing approaches such as the combinatorial

testing approach discussed in Bozic et al. [54], and symbolic execution testing explained in Pham et al. [147]. In this category are also mutation-based testing introduced in Shahriar et al. [154] and test case generator tool for PHP introduced in Stivalet et al. [78].

Bozic et al. [54] evaluated in-parameter-order (IPO-Family) algorithms, namely the IPOG and IPOG-F algorithms, for test case generation in web security testing by using a combinatorial testing approach. They validated the proposed approach on vulnerable web applications published in the OWASP Broken Project37 _{and in the Exploit Database}38_.

Pham et al. [147] presented the design and evaluation of the Hercules approach for finding test inputs which can reproduce a given crash. The approach is based on symbolic execution and its distinctive features. The test input generated by their method serves as a witness of the crash. They illustrated the pertinent aspects of the approach using data from the CVE database.

Shahriar et al. [154] introduced an approach called MUTEC, a Mutation-based Testing of Cross Site Scripting. The approach tries to address XSS vulnerabilities related to web- applications that use PHP and JavaScript code to generate dynamic HTML contents. Shahriar et al. proposed 11 mutation operators to force the generation of an adequate test dataset. The proposed operators were validated by using five open source applications having XSS extracted from the OSVDB repository.

Stivalet et al. [78] presented an automated generator of test cases, which are designed to evaluate source code security analyzers. The tool produces PHP programs with most common vulnerabilities embedded in various code complexities. The authors used CWE weakness dataset and OWASP vulnerabilities categories to generate selected PHP programs test cases. The generated PHP test cases were added to the Software Assurance Reference Dataset (SARD).

37 _{https://www.owasp.org/index.php/OWASP_Broken_Web_Applications_Project}