Automated Testing with Python
assertEqual(code.state(), ’happy’)
Martin Pitt
Why automated tests?
•
avoid regressions
•
easy code changes/refactoring
•
simplify integration
•
design tool
Unit test from Apport
$ python problem_report.py -v
basic creation and operation. ... ok
writing and re-decoding a big random file. ... ok
handling of CompressedValue values. ... ok
reading a report with binary data. ... ok
write_mime() with key filters. ... ok
[...]
---Ran 25 tests in 4.889s
Unit tests
•
smallest possible testable portion of the code
•
noninteractive
•
independent from each other
•
no assumptions about environment or privileges
Integration test from pkgbinarymangler
$ test/run
-v
Installed-Size gets updated ... ok
language packs are not stripped ... ok
OEM PPA for main package ... ok
OEM PPA for main package for blacklisted project ... FAIL
[...]
==========================================================
FAIL: OEM PPA for main package for blacklisted project
---Traceback (most recent call last):
File "test/run", line 379, in test_ppa_oem_main_blacklisted
self.assert_(’locale/fr/LC_MESSAGES/vanilla.mo’ in out)
AssertionError
---Ran 12 tests in 91.783s
Integration tests
•
end-to-end testing of larger scenarios
•
more expensive
•
reasonable assumptions about environment
Python
unittest
structure
1 i m p o r t c o u n t e r 2 i m p o r t u n i t t e s t 3 4 c l a s s C o u n t e r T e s t ( u n i t t e s t . T e s t C a s e ) : 5 d e f s e t U p ( s e l f ) : 6 s e l f . c = c o u n t e r . C o u n t e r ( ) 7 s e l f . w o r k d i r = t e m p f i l e . mkdtemp ( ) 8 9 d e f te a rD ow n ( s e l f ) : 10 s h u t i l . r m t r e e ( s e l f . w o r k d i r ) 11 12 d e f t e s t f o o ( s e l f ) : 13 # . . . 14 15 # main 16 u n i t t e s t . main ( )Python
unittest
test cases
1 d e f t e s t i n i t ( s e l f ) : 2 ’ ’ ’ C o u n t e r i s i n i t i a l i z e d w i t h 0 ’ ’ ’ 3 s e l f . a s s e r t E q u a l ( s e l f . c . c o u n t ( ) , 0 ) 4 5 d e f t e s t i n c ( s e l f ) : 6 ’ ’ ’ C o u n t e r . i n c ( ) a d d s +1 ’ ’ ’ 7 s e l f . c . i n c ( ) 8 s e l f . a s s e r t E q u a l ( s e l f . c . c o u n t ( ) , 1 ) 9 10 d e f t e s t s t r ( s e l f ) : 11 ’ ’ ’ C o u n t e r s t r i n g f o r m a t t i n g ’ ’ ’ 12 s e l f . c . s e t ( 2 3 ) 13 s e l f . a s s e r t E q u a l ( ’ x%s y ’ % s e l f . c , ’ x 2 3 y ’ )Errors and corner cases
1 d e f t e s t n o n i n t ( s e l f ) : 2 ’ ’ ’ R e j e c t s non−i n t e g e r s ’ ’ ’ 3 s e l f . a s s e r t R a i s e s ( T y p e E r r o r , s e l f . c . s e t , 1 . 5 ) 4 5 d e f t e s t h u g e ( s e l f ) : 6 ’ ’ ’ S u p p o r t s a r b i t r a r i l y l a r g e n u m b e r s ’ ’ ’ 7 s e l f . c . s e t ( s y s . m a x i n t ) 8 s e l f . c . i n c ( ) 9 s e l f . a s s e r t ( s e l f . c . c o u n t ( ) > s y s . m a x i n t ) 10 s e l f . a s s e r t R a i s e s ( T y p e E r r o r , s e l f . c . s e t , 1 . 5 ) 11 12 d e f t e s t n o n n e g ( s e l f ) : 13 ’ ’ ’ Can ’ t c o u n t b e l o w z e r o ’ ’ ’ 14 s e l f . c . i n c ( ) 15 s e l f . c . d e c ( ) # s h o u l d be 0 now 16 s e l f . a s s e r t R a i s e s ( V a l u e E r r o r , s e l f . c . d e c ) 17 # d e f i n e d v a l u e a f t e r e r r o r 18 s e l f . a s s e r t E q u a l ( s e l f . c . c o u n t ( ) , 0 )Test-friendly code
•
avoid hardcoded external addresses
•
use proper abstractions (dbapi2, complete URLs, logic vs.
GUI)
Techniques
•
JFDI!
•
locality for unit tests: mock objects
•
locality for integration tests:
files:
mini-chroots with
mkdtemp()
databases:
sqlite :memory:
network:
SimpleHttpServer
on
localhost
packages:
$APT CONFIG
devices:
loop,
$SYSFS PATH
doctest
1 d e f f a c t o r i a l ( n ) : 2 ’ ’ ’ R e t u r n t h e f a c t o r i a l o f n , an e x a c t i n t e g e r >= 0 . 3 4 I f t h e r e s u l t i s s m a l l e n o u g h t o f i t i n an i n t , 5 r e t u r n an i n t . E l s e r e t u r n a l o n g . 6 7 >>> f a c t o r i a l ( 0 ) 8 1 9 >>> f a c t o r i a l ( 5 ) 10 120 11 12 N e g a t i v e v a l u e s a r e n o t a l l o w e d : 13 14 >>> f a c t o r i a l (−1) 15 T r a c e b a c k ( most r e c e n t c a l l l a s t ) : 16 . . . 17 V a l u e E r r o r : n must be >= 0 18 ’ ’ ’ 19 20 n = 0 # . . .Need more power?
•
Mock objects:
python-mock
•
Comprehensive UI testing:
mago
References
•
http://en.wikipedia.org/wiki/Unit testing
•
http://docs.python.org/library/unittest.html
