Multiple Targets and Different Configurations

The number of configurations for MyFuzzer is large and the number of potential targets is also large. To validate the results for this project it is a good idea to run MyFuzzer in the same or different configuration on new targets.

Chapter 7

Conclusion

The results and analysis presented in the experiments and discussion chapters can be used to answer the research questions given in the introduction 1. Each sub- question will be answered here:

How does MyFuzzer compare to existing fuzzers in terms of finding vulnerabili- ties and code coverage:

As can be seen in the results of experiment 1 and 2, MyFuzzer instances perform usually better than Sulley in terms of coverage. In terms of vulnerabilities no signif- icant difference can be seen between Sulley and MyFuzzer except for the fact that MyFuzzer finds vulnerabilities quicker. In Vuln server 1 and 2 Sulley does not find the string format vulnerability. On the other hand MyFuzzer does not always find the same vulnerabilities for EasyFTP than Sulley.

How does the choice of metrics for the fitness function influence the accuracy of MyFuzzer:

In experiment 3 four fuzzers with different fitness functions were tested. The first fit- ness function focused on finding new basic blocks, the second on finding new edges, the third gave both new edges and new basic blocks equal importance and the last one gave the impact of a test-case, new basic blocks and new edges equal impor- tance. From experiment 3 it was learned that most of the time a full focus on new basic blocks delivers the best results for basic block and edge coverage. Combina- tions of multiple metrics generally performed less. In terms of finding vulnerabilities no significant difference could be observed. In general the fuzzers for experiment 3 found the same set of vulnerabilities.

How does the choice of fuzzing strategy influence MyFuzzer in terms of finding vulnerabilities and code coverage: The last subquestion is not the most important

and cannot be answered with the most confidence. In general strategy 1 seems to be the best strategy for MyFuzzer, which was enough incentive to use strategy 1 as the main strategy.

In conclusion, this report has shown two things. First of all it has shown that it is 103

possible to create a fuzzer driven by a genetic algorithm without requiring access to the source-code. Secondly the research part of this project confirms that a fuzzer with a genetic algorithm improves over a fuzzer without a genetic algorithm. The research also indicates that a genetic algorithm which tries to maximize the number of basic blocks tested gives the best results.

Additional experiments with different types of targets and different parameter set- tings can be done to validate the results obtained in this project.

Bibliography

[1] Fuzzing. https://www.owasp.org/index.php/Fuzzing, 2016. [Online; ac-

cessed 04-January-2017].

[2] Dynamorio. http://dynamorio.org, 2017. [Online; accessed 19-April-2017].

[3] Peach fuzzer. http://www.peachfuzzer.com/, 2017. [Online; accessed 25-

January-2017].

[4] B. H. Arabi. Solving np-complete problems using genetic algorithms. In 2016 UKSim-AMSS 18th International Conference on Computer Modelling and Sim- ulation (UKSim), pages 43–48, April 2016.

[5] S. Bekrar, C. Bekrar, R. Groz, and L. Mounier. Finding software vulnerabilities by smart fuzzing. In 2011 Fourth IEEE International Conference on Software Testing, Verification and Validation, pages 427–430, March 2011.

[6] cacalabs. zzuf. http://caca.zoy.org/wiki/zzuf, 2016. [Online; accessed

25-January-2017].

[7] J. Cai, S. Yang, J. Men, and J. He. Automatic software vulnerability detection based on guided deep fuzzing. In 2014 IEEE 5th International Conference on Software Engineering and Service Science, pages 231–234, June 2014.

[8] J. Cai, P. Zou, D. Xiong, and J. He. A guided fuzzing approach for security testing of network protocol software. In2015 6th IEEE International Conference on Software Engineering and Service Science (ICSESS), pages 726–729, Sept

2015.

[9] Steve Cornett. Code coverage analysis.http://www.bullseye.com/coverage. html, 2014. [Online; accessed 25-January-2017].

[10] J. Demott, R.J. Enbody, and W. Punch. Revolutionizing the field of grey-box attack surface testing with evolutionary fuzzing. Oct 2007.

[11] Parul Garg. Fuzzing - mutation vs. generation. http://resources. infosecinstitute.com/fuzzing-mutation-vs-generation, 2012. [Online;

accessed 03-January-2017].

[12] Dan Geer. The physics of digital law. InCyberCrime and Digital Law Enforce- ment Conference, March 2004.

[13] Intel. Pin - a dynamic binary instrumentation tool. https://software.intel. com/en-us/articles/pin-a-dynamic-binary-instrumentation-tool, 2017.

[Online; accessed 19-April-2017].

[14] E. Jskel. Genetic algorithm in code coverage guided fuzz testing. Master’s thesis, Dec 2015.

[15] Richard Kissel. Glossary of key information security terms. Technical report, National Institute of Standards and Technology, 2013.

[16] lcamtuf. american fuzzy lop.http://lcamtuf.coredump.cx/afl, 2016. [Online;

accessed 05-January-2017].

[17] Roger Lee, Accepted For The Council, Carolyn R. Hodges, and Roger Lee Seagle. A framework for file format fuzzing with genetic algorithms. 2012. [18] G. H. Liu, G. Wu, Z. Tao, J. M. Shuai, and Z. C. Tang. Vulnerability anal-

ysis for x86 executables using genetic algorithm and fuzzing. In 2008 Third International Conference on Convergence and Hybrid Information Technology,

volume 2, pages 491–497, Nov 2008.

[19] Mauro Pezz Luciano Baresi. An introduction to software testing. Electronic Notes in Theoretical Computer Science, 2006.

[20] Gary McGraw. Software Security: Building Security In. Addison-Wesley Pro-

fessional., first edition, 2006.

[21] Richard Mcnally, Ken Yiu, Duncan Grove, and Damien Gerhardy. Fuzzing: The state of the art. 2017.

[22] Ruth Breu Matthias Buchler Alexander Pretschner Michael Felderer, Philipp Zech. Model-based security testing: a taxonomy and systematic classi- fication. Software Testing, Verification and Reliability, 2016.

[23] Melanie Mitchell. An Introduction to Genetic Algorithms. The MIT Press, first

BIBLIOGRAPHY 107

[24] NIST. National vulnerability database. https://web.nvd.nist.gov/view/ vuln/statistics-results?adv_search=true&cves=on&cvss_version=3. [On-

line; accessed 25-January-2017.

[25] Jiantao Pan. Software testing. _{https://users.ece.cmu.edu/~koopman/des_}

s99/sw_testing/, 1999. [Online; accessed 25-January-2017].

[26] Jeff Offutt Paul Amman.Introduction to Software Testing. Cambridge University

Press, second edition, 2008.

[27] Ryan Sears Pedram Amini, Aaron Portnoy. Fuzzing - application and file fuzzing. https://github.com/OpenRCE/sulley, 2016. [Online; accessed 03-

January-2017].

[28] Tsong Yueh Chen John Clark Myra B. Cohen Wolfgang Grieskamp Mark Har- man Mary Jean Harrold Phil McMinn Antonia Bertolino J. Jenny Li Hong Zhu Saswat Anand, Edmund K. Burke. An orchestrated survey of methodologies for automated software test case generation. Journal of Systems and Software,

86(8):1978 – 2001, 2013.

[29] Bruce Schneier. Complexity the worst enemy of security. https://www. schneier.com/news/archives/2012/12/complexity_the_worst.html, 2012.

[Online; accessed 19-April-2017].

[30] Offensive Security. Offensive securitys exploit database archive.https://www. exploit-db.com/. [Online; accessed 14-October-2017.

[31] B. Shuai, M. Li, H. Li, Q. Zhang, and C. Tang. Software vulnerability detection using genetic algorithm and dynamic taint analysis. In 2013 3rd International Conference on Consumer Electronics, Communications and Networks, pages

589–593, Nov 2013.

[32] Huang Song, Wang Liang, Zheng Changyou, and H. Yu. A software security testing method based on typical defects. In 2010 International Conference on Computer Application and System Modeling (ICCASM 2010), volume 5, pages

V5–150–V5–153, Oct 2010.

[33] Wikipedia. Genetic algorithm. https://en.wikipedia.org/wiki/Genetic_ algorithm. [Online; accessed 18-April-2017].