Can I get the box without having that box in storage (database)? No
2. The risk of releasing untested code to our users, because version 0 (which was supposed to be tested) wasn't tested
As you can see, negligence in checking the version of the code to be tested can have very negative consequences.
My advice to you is this: If you screw up and test version 1.0 instead of version 2.0, immediately
acknowledge your fault. Everyone makes mistakes. It's life. Just forget about your plans for the evening, weekend, or holiday, and stay in the office until you finish testing version 2.0.
BTW
It's a good idea for a start-up employee to have a sleeping bag, a change of clothes, a towel, and some personal hygiene items ready in the office – there is always the chance for some wild overtime, and you don't want to drive after fifteen hours of work.
Sad things, like testing of wrong version, take place because of a tester's negligence, but sometimes other factors, like illogical names of test environments, contribute to the situation. Here’s how to prevent this:
1. Ask your release engineer how to find out the current code version in the test environment and apply this knowledge EVERY time before testing.
2. Ask your IT engineer about choosing logical names for test environments. An IT person can enjoy whatever he or she likes (for instance, mountain climbing), but it's difficult to understand the difference
between Everest and Elbrus when talking about a version of software. Let's create two test environments instead:
http://old.sharelane.com for the application version which is currently on our live site:
http://www.sharelane.com. So, if a bug is released to our users, we have http://old.sharelane.com to test a bug fix before releasing that bug fix to the live site. Between ourselves we can call this test
environment "Old".
http://main.sharelane.com for the application version of the coming release; the testers do testing on http://main.sharelane.com, and when we are ready for the release, the live site,
http://www.sharelane.com, will get the version of the software accepted for release on http://main.sharelane.com. Between ourselves we can call this test environment "Main".
3. If the current way of finding out the version of the code and the DB is not convenient (e.g., you always have to go and ask the release engineer about those versions), suggest that he or she implements one of the following:
a. Add a Web page with information about - the release version
- the build number (more about this in a minute) - the DB schema version (more about this in a minute);
SL
For example, if this is implemented, you can go to the page: http://main.sharelane.com/version.txt and see something like this:Release version: 1.0
Build number: 23
DB schema version: 34
Important: the "version of the code" or the "version of test environment" consists of the three components above. Remember to check for these three things every time before you start testing. We'll elaborate more about each of the components soon.
b. Another good approach is to include information about the current version as a comment written into the html code of each Web page.
Example
SL
Do this:1. Go to main.sharelane.com
2. Point your cursor on an object-free spot of the Web page; an "object-free spot" is any spot on the Web page without a Web object (such as an image, link, etc.).
3. Do a single right click.
4. Select the "View Source" (IE) or "View Page Source" (Firefox) option 5. Find the version number.
For example:
<!-- application version 1.0-23/34 -- >
You can find out application version of ShareLane looking into HTML of any Web page on http://main.sharelane.com (exception is Test Portal pages).
BTW
From now on, let’s call the full version of our code the “application version”.
At ShareLane, we specify application version in this format:
<major release number*>.<minor release number>-<build number>/<DB version number>
* we’ll talk about release types later.
Three last things about "Coding" stage:
First: During the “Coding” stage, programmers write code for the coming release and testers write test cases to test that code. I recommend that files with test suites, just like files with specs, should be
- stored in CVS (or other version control system);
- available to anyone inside the company.
The main advantages of storing files with test suites in CVS (or other version control system) are:
- it eliminates the risk of an accidental deletion of a file;
- it allows access to old editions of the file and its history;
- the file is stored on the shared server, and everyone who needs to (and who has a right to do so) can take that file to
> execute, modify, or delete existing test cases;
> append new test cases.
Test Case Reviews
Second: It’s very good idea to have a tradition of test case review meetings. Test case review meetings take place before the stage “Testing and bug fixes” and right after the tester has finished writing test cases. What happens is that the following people:
- the PM who wrote the spec,
- the programmer who is responsible for coding that spec and - the tester who has finished test cases to test that code
get together, and the tester makes a mini presentation about the way he or she is going to do testing.
Before the meeting I usually make hard copy of the test suite for everyone who accepted my meeting invitation. During the meeting I give a short overview of each test case, and everyone can see what I’m talking about.
The great value of this kind of meeting is that in many cases PMs and programmers
- give testers new IDEAs for testing and/or - find omissions and/or errors in test cases.
BTW
After the test case review meeting, send out an email to every participant and to those who were invited but couldn’t come to the meeting. In this email, list all improvements that you’ve talked about during your meeting. This email is very useful because
- you’ll refresh the suggested ideas in your brain;
- you’ll give others an opportunity to see that you understood them correctly.
In many large Internet companies, the test case review meeting is an obligatory thing.
Third: After my wife read about UI and logical bugs, she asked me a simple question: If an HTML link leads to the wrong page, is it a UI or a logical bug?
Interesting question:
- on the one hand, it’s about presentation;
- on the other hand, sometimes links are dynamically generated by our code (e.g., the link “log out” is generated only if user is logged in).
So, my response was:
- “If the misleading link is hard-coded into the HTML code of the Web page, then it’s a UI bug, because there is no problem with processing here and the presentation is wrong."
- “If the misleading link is dynamically generated by our code, then it’s both a logical and a UI bug, because there is a definite issue with processing and there is definite issue with presentation.”
Does this make sense? What do you think?
That’s all for the “Coding” stage, my friends. I hope that you’ve learned some valuable stuff that you will be able to use in your companies.
Testing And Bug Fixes
We are going to cover a lot about this stage during the rest of our Course. So, for now, let’s be as laconic as Spartans.
After the developers have finished writing the code, the release engineers make it available for the testers in the test environment (http://main.sharelane.com). The testers perform a quick smoke test (also called a sanity test or confidence test) to check if the software is testable.
Example
If we cannot log into the account on main.sharelane.com, we simply cannot proceed with test execution, i.e., the software is not testable.
As a rule, a smoke test doesn’t take more than half an hour, and it usually doesn’t require any special test cases.
If the smoke test fails, we communicate this to the programmers and release engineers (there can be an error on the release engineering side), and they work to fix the problem. Once the problem is fixed, we perform the smoke test again. This mini cycle takes place until the smoke test passes. Once the smoke test passes, the release engineers freeze the code and push it to the test environment
(main.sharelane.com), and the testers start the new feature testing by executing test cases written for this release.
Brain positioning
“Feature” is a much wider term than “functionality” which we’ve been using so far. Functionality is the ability to accomplish some task. For example, the functionality (ability) of a bottle opener is to open bottles.
Depending on the context, the term "feature” means - the ability to accomplish some task or - some characteristic of the software.
For example,
- the Shopping Cart can be called both a "feature" and "functionality", because it is an ability that allows users to store items they want to buy.
- the color of the Shopping Cart link can be called only a "feature", because the color of that link
>is a characteristic of the software and
>it does not have functional (i.e., task-solving) aspects.
When in doubt, use the term “feature.”
BTW
A favorite developer's expression, "It's not a bug, it's a feature", in human language sounds like, "That something is not a problem with my code. It works (and/or looks) exactly as I want."
After the new feature testing is finished, the testers start to test the old features. This type of testing is called regression testing. Regression testing is performed to check if code modifications made for this release have broken something in the old features. After we are finished with new feature testing for our release 2.0, we are going to use our test cases written for release 1.0 for regression testing.
BTW
Does this mean that during regression testing for release 25.0, we have to execute all test cases from releases 1.0 to 24.0 inclusively?
Well, this is one of the most difficult questions in testing, and we’ll address it later.
Found bugs are filed into the bug tracking system, the programmers fix them, and the testers verify those fixes.
Once the regression testing is finished (there is usually a concrete deadline), testers perform an acceptance testing (also called a certification testing). During acceptance testing, the testers:
- execute a special set of P1 test cases (often in form of checklists)
- do ad hoc testing (undocumented testing based on intuition and inspiration)
Sometimes, when a young start-up has an important release (e.g., its first release), the whole company stays overnight to do acceptance testing.
Once the acceptance testing is finished, a gang of brothers and sisters (PMs, testers, developers, release engineers, managers) get together for a Go/No-Go meeting. At this Go/No-Go meeting the group decides if the code is ready to be released to the production environment (also called “production,”
“prod,” or "live") so the users can enjoy the results of the team’s efforts. If it's a "Go," the release engineers release the code to prod. If it's a "No-Go," the responsible parties stop eating and sleeping until the issues are resolved. Once all the issues are resolved, there is another Go/No-Go meeting, and so on … until release to prod takes place.
Release
Example
In the spirit of Steven King’s novel, a little boy – a dreamer, a book lover, and an insect collector – is being constantly humiliated by his siblings, classmates, and accidental bystanders. One day he says,
“Enough is enough,” and starts to cut, shoot, strangle, and burn his abusers, and for the sake of
prevention, all others who are in his way. This situation is about letting the steam out – or “release” in everyday terms.
Luckily for us, in the software industry the term “release” is used in a totally different way:
- as a verb, “to release” means to transfer a piece of software to the users. For example, we can ask a release engineer to release the code to production.
- as a noun, “release” means a certain piece of software. For example, we can say: “We are testing release 5.0.”
BTW
We’ll apply the term “production version” or “version of prod” to the release that is in production now.
We’ll apply terms “coming release” to whatever version we are going to release next.
The important thing to understand is that a release is not some kind of abstract software; it is a package of concrete files having concrete versions. Please pay attention.
Example
The release 5.0 contains 63 files. Each of these 63 files has its own version. When the testers have finished testing release 5.0, the release engineer will release precisely the same 63 files that were in the testing environment when the testers finished testing. If the testers finish the acceptance test on January 03 at 4:00 pm, and at that time the file register.py had its own version 5.34, then it must be exactly version 5.34 of register.py that is included into the release 5.0 when 5.0 goes to production.
BTW, the version number is automatically assigned to file by CVS (or whatever version control system is used) every time a developer saves (or more professionally, “commits”) updated version of the file to CVS.
The purpose of the release is to transfer one or a combination of the following things, to production: