Test automation is a cornerstone of a true CI/CD pipeline; however, it’s a very complicated topic. The tools vary by the language in which the application is written, the nature of the application itself, and even the composition of the team or teams writing the software. Of all the problems in the CI/CD space, this is the most challenging one. Not only is it a challenge to decide what to test, it’s difficult to determine how best to test it. There are unit tests, integration tests, functional tests, system tests, validation tests, regression tests, black box testing, white box testing, static code analysis, dynamic code analysis and even open-source license, compliance and security analysis. The list could continue, opening this space up to
DEEP DIVE
become another seemingly endless array of choices. In the end though, it is best to stay focused on answering the question, “Is this code ready to be used?” and then come up with your own organization’s definition of “ready”. That will help you make decisions about what kinds of testing you should be doing now, later and perhaps not at all. As you journey down the road of test automation, your definition of “ready” likely will become more and more strict, and you’ll iteratively bring on additional tools to meet the evolving criteria. The most common classes of automated testing, unit testing, system testing and functional testing are all great places to start. They all have lots of good FOSS options available.
Hundreds of different unit test frameworks are available, with at least a handful for nearly every language that has seen any amount of real world use. There is an incredible list of these frameworks available at Wikipedia. Start your search for a tool there, or search for “unit testing for $my_language” in your search engine of choice, and choose one that seems to be actively used and developed, and one that can be made to integrate with the other tools you intend to use. Many of them are “xUnit” style, which is a very common model for unit testing. If you choose one of that type, it’s more likely that your developers will be comfortable writing tests for it, and there’s a good chance it will create a results report in a JUnit-compatible XML file. JUnit XML reports are the lingua franca of the unit test world, and having reports in that format makes it far more likely that whatever tool you want to record your results in will be able to parse the report.
System testing isn’t quite so tightly defined. Here again is a wide-open problem space with a multitude of possible solutions that will be heavily influenced by your particular situation. My preferred starting approach is fairly repeatable and broadly applicable. Deploy a disposable instance of your application (usually in containers) and run a load test with a tool like Gatling or Postman to run through the core functionality of your app quickly. If those tests fail, there’s a good chance you have an system issue. Postman itself isn’t a FOSS tool, but it is free for most uses, and the Postman folks release a lot of supporting tools as FOSS and generally seem to be a good FOSS community members.
DEEP DIVE
It’s also worth noting that system testing is often mistakenly called “integration testing”. Integration testing is a step that would traditionally come between unit testing and system testing, but one should consider skipping it early in the adoption of automated testing practices, as it usually provides tangible value only in very complicated software written by very large teams or composed of the work from several separate teams.
For functional testing, the standout FOSS tool is Selenium, which forms the core of many other testing automation tools, both FOSS and commercial. If your application exposes anything through a webpage, Selenium or something like it should be in your toolkit.
And finally, all the testing in the world doesn’t mean much if you can’t view the results. Jenkins can display test results itself, but running Sonarqube adds a lot of value. Not only does it give you a great view of how your test results have changed over time, it can perform various kinds of static analysis on your code if you are using a supported language. It’s another Open-Core-licensed tool, and some very useful features have been moved into the commercial version recently—perhaps most notably the ability to track multiple branches of a single codebase easily.
Conclusion
One could use a selection of the tools from each of the lanes listed above and provide the framework for an effective CI/CD pipeline. The options here are only a few of the possible choices; however, they are ones with a proven record of delivering value. And ultimately, that’s the point: getting to value. In the end, that’s what a CI/CD pipeline is for, delivering value to your users as quickly and smoothly as possible, by reducing friction within your development and deployment process. And that is an effective early step in embracing DevOps. ◾
Quentin Hartman is a lifelong technology enthusiast who has been working in one aspect or another of network and system
administration for more than 20 years. The vast majority of that time has been spent using Linux and other FOSS tools to help meet the needs of organizations large and small. Quentin is currently working as the Director of Infrastructure and DevOps at Finalze where he helps the team build great software with a soul. He lives near Denver, Colorado, with his wife, three daughters and five chickens. He can be reached as “qhartman” on twitter.com and keybase.io.
DEEP DIVE
Send comments or feedback
via http://www.linuxjournal.com/contact or email [email protected].
Resources
• Open Source Business Models Considered Harmful
• DevOps Bookmarks
• GitLab Core
• GitLab Features Matrix
• Gitea
• Jenkins
• CloudBees
• Jenkins Blue Ocean
• Drone
• List of Unit Testing Frameworks (Wikipedia)
• Gatling
• Postman
• Overview of Integration Testing (Wikipedia)
• Selenium
• Sonarqube
The largest Open Source | Open Tech | Open Web conference on the U.S. east coast
More than 4,000 technologists expected
from all over the world
Nearly 250
sessions More than 240 speakers
Priced affordably to encourage and
enable access
O C T O B E R 1 3 - 1 5 , 2 0 1 9
R A L E I G H , N C U S A
R E G I S T R AT I O N I S J U S T $ 1 7 9 T H RO U G H AU G U S T 3 1 E nt e r p ro m o c o d e L i n u x J o u r n a l f o r 2 0 % d i s c o u nt