Intro
Who am I?
Overview
Continuous Integration
The Tools
Selenium Overview
Fitnesse Overview
Data Dependence
My Solution
Demo
What is CI?
From Wikipedia:
Continuous Integration (CI) implements a continuous processes of applying quality control - small pieces of effort, applied frequently. Continuous integration aims to improve the quality of software, and to reduce the time taken to deliver it, by replacing the traditional practice of applying quality control after completing all development.
In English:
automatic code integration and validation!
What do we need?
A Build Server to control everything and store reporting
A development/deployment process that supports being automated.
Scripts to do the work
One or more test frameworks to test the code (nUnit, Fitnesse, Selenium)
A minimal set of tests that cover a broad scope of functionality
What do we get?
Automatic testing and validation for each code change
Immediate and automatic isolation of the cause
Up-to-date, centralized reporting
You know who broke it, and who needs to fix it.
You never will test a broken environment again.
The Tools
Build Server (Hudson) Runs our Tests
Unit Tests
Tests the individual functions
Fitnesse Tests:
Database (DB Fit)
API calls
Web Services
Selenium Tests:
UI Issues
Functionality problems
Javascript Errors
Selenium IDE
Firefox only plug- in, no other
browsers.
Step based testing, no complex functionality
Record and Playback
Fast test prototyping
Troubleshooting
Can be run from Hudson using a plug-in.
Selenium Core
Stored on your website’s web server
Executable via a url
Supports any browser
Tests stored in a central repository
Has problems supporting https or changing domains
Selenium RC
Client/server java application needs to be installed
Supports any Browser/OS combination
Test can be run locally or remotely
Controlled and executed through xUnit
Test written in your favorite programming language/IDE (C#, java, python, ruby)
Access to full features of that language and any extra libraries
[Test]
public void testGoogle() {
selenium.Open(“http://www.google.com/");
selenium.waitForPageToLoad(“30000”);
selenium.Type("q", "selenium");
selenium.Click("btnG");
selenium.waitForPageToLoad(“30000”);
Assert.IsTrue(selenium.IsTextPresent("Selenium web application testing system"));
}
Selenium Grid
Distributed parallel testing
More setup, but faster execution
Run on multiple machines
simultaneously
Generates more load
Pros Cons
Injects itself as javascript into the page object
Any browser that supports javascript supports selenium
The ONLY tool that can work with any browser/os
Extremely fast execution time
Direct access to page objects
Doesn’t hijack your computer
You can do other things while its running
Full access to outside libraries
If you need something else, you can code it.
Pick your language
Record in IDE and convert to RC.
Easier for non-coders
Fast test prototyping
Troubleshooting
Open source tools can be customized
Injects itself as javascript into the page object
Non Firefox browsers don’t always work perfectly
It can’t see anything outside of the page
Can’t click on the menu, navigation buttons, or any extra windows
Predefined routines reproduce this functionality
XSS Limitations
Changing domains/https
It “fakes” a lot
It doesn’t use the mouse/keyboard
Doesn’t really click on anything
Accesses each item directly and calls a function
Can bypass front end functionality such as
“onclick”
Can access hidden items even if you don’t want it to
Some functionality can’t be automated
But can be gotten around
Open source tools require a lot of coding
How Selenium Works
Fitnesse Overview
What is Fitnesse?
Fitnesse = FIT + Wiki
FIT : Framework for Integrated Testing - The engine that translates tables into actions
Wiki: The GUI that allows the user to edit and run tests
A website that allows us to create, edit, and run tests.
The tests must be written as a table
Each row is a test
Fitnesse continued
Typically used to test an application or database directly, without accessing the GUI.
Calls a Web service, API, DB Procedure
Test the Business Logic directly.
We don’t have to try to “trick” the ui into an incorrect state
Typically more stable than GUI testing as less functionality changes over time
Requires the creation of a Fixture which knows how to translate the test into an action.
The fixtures are many times written by the application developers
Some “standard” fixtures exist, such as the RESTFixture for testing REST based services
Once written, the fixtures can be largely ignored
Different fixtures allow different functionality.
Fitnesse continued
Typical data driven testing - same test multiple times for all data types
Boundary conditions
Error code checking
Business Logic
Acceptance Testing-
Wiki style input allows non-coders to develop and run tests.
Business and Product personnel can (theoretically) write acceptance tests
However it requires a knowledge of the AUT
GUI testing is possible with appropriate fixture
Can’t test for javascript errors without a GUI test
However selenium is more robust
Fitnesse
Fitnesse pros and cons
Tests are editable and executable from a web page.
Uses Xpath.
Can generate fake bugs (stuff looks broken because of a Fitnesse problem)
Test maintenance can be time consuming
Everything is a text document
Same tests many different places
Global variables allow you to change url, user, etc very easily
Many tests can require unique information (id, name) that can change in different environments.
Run-time variables allow you to create more dynamic tests.
Knowing what data should be returned can be troublesome
Can generate exact error-causing conditions
Useless if your application sucks
Data Dependence
The biggest problem with automation is changing data
Time changes, items expire, dates change.
Need to test in different environments going to out to production.
Many items are referenced by an ID that is unique, or increment.
Need to know what are the expected results of your test.
If something you assume is incorrect, your tests will fail.
User not there
Items expired
Dates changed
GUI item changed or vanished
In unit testing, you can mock data to test with.
When testing a real system, your data can fluctuate
DB Refresh
Virtualized environments
User activity can change your data set
Date oriented things expire
Id’s increment
Using external or live data
Data Options
Set our data manually each run.
DB Copy/refresh to return to a known state.
Start from and return to a known state
CRUD only your data
Get current data
Connect to the db to grab data.
Run UI automation to get data.
Generate data
Use automation to generate a data set at run time
My Solution
Data Generation Suite-
Can point to any client/environment.
Checks to see if my data is there, if it is it deletes it.
Then creates all data fresh, and sets state.
A Smoke Test to be run by the Hudson server.
Checks all major functionality.
Checks each of the functional areas
Doesn’t validate what data it gets. (it fluctuates too much)
A Regression Test that tests full functionality.
Data Generation Suite needs to be run first
Tests full functionality and validates data is correct.
Fitnesse tests that check the services
Use the data selenium created.
Verify boundary conditions
Verify response code and format.
Verify errors
The end result
All builds automatically are smoke tested for viability.
Takes ~15 minutes to regression test a build in each environment
Can change users, clients, environments, configuration.
All tests don’t always pass in all environments, have to check test results.
Can use Test Driven Development methodology
Create tests first
