Software Testing with Python

Software Testing with Python

Magnus Lyckå Thinkware AB www.thinkware.se

EuroPython Conference 2004 Chalmers, Göteborg, Sweden

In the next 30 minutes you should...

● Learn about different aspects of software testing

such as unit tests and acceptance tests.

● See how the standard modules for unit tests are

used, learn about some alternatives and have a grasp of their respective pros and cons.

● Know about some options for acceptance tests. ● Know about some standard Python modules and

third party tools that are helpful in software testing and related activities.

What is software testing?

● “Testing means verifying that your code is

running correctly by exercising the code under known conditions and checking that the results are as expected.” – Alex Martelli, Python in a Nutshell

● If we are serious with our requirements on a

software system, we should make sure that we

verify them, and testing is the most common kind of verification for software.

Software testing in a context

Software Quality Assurance

Software Verification

Automating tests

● Writing automated tests is usually more work

than testing things manually once, but they make it much easier to...

– Work in a repeatable and predicable way – Run tests more often

– Run tests unattended – Find regression bugs

– Test more operating systems

Example of requirements and

tests on different levels

System Specification Acceptance Test

Detailed Design API Specification

Unit Test

Requirements and tests on different

levels in Extreme Programming (XP)

System specified through customer tests

Design by writing API validated through

programmer tests

Consequences of the XP approach

● Requirements are written in a verifiable way.

● No discrepancies between requirements and tests. ● We get unambiguous and repeatable ways of

verifying requirements.

● Continuous picture of project progress

But...

● Customers must be able to understand the tests. ● We need test automation frameworks.

Software Testing with Python

Unit tests in Python

● Unit tests, or programmer tests, are tests the

programmer writes to make sure that the code does what the programmer intended it to do.

● Unit tests can also help document how a piece of

code (e.g. a class) is supposed to be used.

● Python has two standard modules for unit testing,

unittest and doctest.

● There are several third party modules that aim to

The

_unittest

module

● The Python unittest module is modeled after

the unit test modules developed within the XP community by Kent Beck and Erich Gamma.

● It's intended for a test-first approach, where tests

are written before the actual code that it tests.

● Always writing tests first is a bit like always

eating your meat before you eat your dessert...

● When your tests pass, you're done. (XPers

Concepts in

_unittest

● A test fixture consists of the actions needed to

setup for a test, and clean up afterwards.

● A test case is the smallest unit of testing. It might

for instance call a function and check the results.

● A test suite is a collection of test cases and/or test

suites that should be executed together.

● A test runner runs test suites (or individual test

cases), collects the results and presents it as text or graphically to the user.

The

_TestCase

class

● Test cases sharing a common fixture will be

implemented as methods (whose names start with

test) in a sub class of unittest.TestCase.

● The test runner will create one instance object per

test case, and run SetUp, followed by the test case method and finally tearDown.

● Checks are made with the methods fail, failIf,

assert_/failUnless, assertEqual/failUnlessEqual, assertNotEqual/failIfEqual and

The test runner

● In many cases, running tests is as simple as

executing _{unitest.main()} in the file containing the test cases.

● Testing progress will be reported, and the test

runner makes a distinction between FAILURE and ERROR.

– FAILURE means that the check didn't produce the

expected result.

– ERROR means that something else in the test went

Unittest template

import unittest import MyModule

class MyModuleTest(unittest.TestCase):

def testOneCase(self):

pass

if __name__ == '__main__':

unittest.main()

Unittest examples

● divtest.py

– Trivial example to demonstrate features of unittest.

● container_ut.py

– Real world example from SystemSpecifyer (see

The

_doctest

module

● The Python doctest module was written by

Tim Peters to check that coding examples in doc-strings were correct, by testing them.

● It has also found use as a more general unit

testing module, and from Python 2.3, this has been made more convenient.

● Using doctest helps you combine tests and

documentation in a more readable way than

Doctest example

"""

Return a divided with b.

Divide will return the result of an integer division >>> divide(6,3)

2

Division by zero is handled as expected... >>> divide(1,0)

Traceback (most recent call last): ...

ZeroDivisionError: integer division or modulo by zero """

result = a / b return result

if __name__ == '__main__': import doctest, sys

Unittest vs doctest

● Unittest mantra: Test a little, code a little

– The logical choice if you use extreme programming

or test-driven development of some other kind.

– Maybe better for complex tests?

● Doctest mantra: Code a little, test a little

– The natural companion if you prefer to experiment

interactively with your classes and functions in the Python interactive environment.

– Probably better for adding tests to already written

Unittest vs doctest on the web

● Charming Python: Testing frameworks in Python

– http://www-106.ibm.com/developerworks/linux/library/l-cptest.html

● Literate Testing: Automated Testing with doctest

– http://www.python.org/pycon/dc2004/papers/4/

● DocTest at WikiWiki

Other contenders...

● Sancho – a unit testing framework from MEMS

Exchange. Adds coverage analysis etc.

– http://www.mems-exchange.org/software/sancho/

● Peckcheck by Darius Bacon.

– http://www.accesscom.com/~darius/software/clickcheck.html

● The “second standard library” std.

Unittest complements

● Pester – the Python version of Jester.

– It finds code that is not covered by tests, makes some

change to your code, runs your tests, and if the tests pass it displays a message saying what it changed.

– http://jester.sourceforge.net/

● Test coverage support

– E.g. http://www.garethrees.org/2001/12/04/python-coverage/

● Mock objects: Here, 9:30

Software Testing with Python

Python Integration Test

● No static linking. We must run code to test

interfaces! (On the other hand, compilers and linkers fail to see lots of integration problems...)

● Since building/linking isn't an issue in a pure

Python project, we can use either unit test tools or acceptance test tools for integration tests.

● Python is good at gluing things together, and thus

helpful in all sorts of integration work. See e.g.

Software Testing with Python

Test Frameworks using Python

● QMTest – Uses Python for test expressions.

– http://www.codesourcery.com/qmtest/

● PyFIT – Python clone of Ward Cunningham's

Framework For Integrated Testing (FIT).

– http://www.xprogramming.com/software.htm

● Software Testing Automation Framework (STAF)

Big framework from IBM with Python API.

– http://staf.sourceforge.net/

Software Testing with Python

The usual suspects...

● Python is excellent for analysis and manipulation

of data. A great tool for test related work.

● The re library is useful but use special tools

when available, such as for parsing XML files.

● For dealing with files you might use the modules

gzip, zipfile, codecs, filecmp, struct, sgmllib, xml.* etc

● Python interfaces well with internet services,

difflib

● More or less like the unix diff utility, but as a

Python module. (New in version 2.1.)

● Useful when we want a more detailed response

than FAILED from a test – particularly if we are trying to spot small changes in big amounts of data.

● Great for spotting differences in configurations,

Difflib

example

>>> diff =d.compare(['Hello World', "This is the same.",

"Time flies like an arrow. Isn't that great?"], ['Hello World!', "This is the same.",

"Fruit flies like a banana. Isn't that great?"]) >>> print "\n".join(diff)

- Hello World + Hello World! ? +

This is the same.

- Time flies like an arrow. Isn't that great? ? ^ ^^ - ^^^^

+ Fruit flies like a banana. Isn't that great? ? ^^^ ^^ +++ ^^

AT&T Graphviz

● Tools to generate graphs from C-like text files.

– Dot – for directed graphs.

– Neato – for undirected graphs

– Fairly clever algorithms for decent looking layout – Generates graphs in many file formats

– Suitable for automatic generation of graphs – Some control over placement is possible

● Great for dependency analysis!

Remember this?

System Specification Acceptance Test

Detailed Design API Specification

Unit Test

AT&T Graphviz example 1

digraph G {

label = "V-model"; sysspec

[label="System Specification"]; systest [label="Acceptance Test"]; apispec [label="API Specification"]; apitest [label="Integration Test"]; unitspec [label="Detailed Design"]; unittest [label="Unit Test"];

sysspec -> apispec -> unitspec; unittest -> apitest -> systest; sysspec -> systest;

apispec -> apitest; unitspec -> unittest; }

AT&T Graphviz example 2

digraph G {

label = "V-model";

node [shape=box, style=filled, color="#CCCCFF"];

sysspec [label=

"System Specification"];

apispec [label="API Specification"]; unitspec [label="Detailed Design"]; node [shape=box, style=filled,

color="#FFFF99"];

systest [label="Acceptance Test"]; apitest [label="Integration Test"]; unittest [label="Unit Test"];

{rank = same; sysspec; systest}; {rank = same; apispec; apitest}; {rank = same; unitspec; unittest}; ...

Useful Books

● Python in a Nutshell, by Alex Martelli

● Text Processing in Python, by David Mertz

● Software Test Automation, by Fewster & Graham ● Just Enough Software Test Automation, by

Daniel J. Mosley & Bruce A. Posey

● Testing Extreme Programming, by Lisa Crispin

and Tip House

● Test-Driven Development By Example, by Kent