Quick and Dirty Software Testing Without a Formal Testing Plan
I like building Test Cases from Use Cases. I do it for fun, sometimes.
Last week however, we were presented with a sudden deadline. We had no forewarning. We were not prepared. The project was dormant for a long time, and out of the blue it was revived as an emergency go-live. I had not seen that happen before. We were in a position where there was no formal testing plan, no test cases, and certainly no Use Cases. All we had was our basic knowledge of the context of the software and the functionality that had been added (this was a version release).
Our development team had not worked on this project. Those developers had long gone, off to greener pastures. They left behind spaghetti code held together with chewing gum and toothpicks that nobody understood or particularly wanted to understand. It was a classic ASP site, and we had moved to .NET some time ago.
No need to lament upon what you do not have, right? What we had was a list of features - contained in sales literature that had been distributed to customers some time ago. A list of features is a list of functionality. One might say that a feature is a Functional Requirement. If you know what the software is supposed to do, you have something to test.
You cannot test without a testing plan, period. Even “Let’s just click stuff and see what breaks” is a crude plan.
If you don’t know how the software is supposed to work, it cannot be tested. You can see what it appears to do, and you can test what it appears to do, but that is almost a self-fulfilling prophecy.
So, in our case, we had a list of features and could extrapolate a list of possible Use Cases. Sadly, there was no time for Use Cases. We had to go live in a few days.
We wrote User Stories. I hate to admit it, but we did. This was a case where the argument for User Stories was clear. They would provide a narrative we could test against. It would not be very finely detailed, and it would not be as thourough as I would have liked, but it would be something to test against.
We started with the basic (high level) functionality. If there were problems at that level, we needed to know ASAP. I think it is important to identify the show stoppers as soon as possible. In a scenario where you are not familiar with the spaghetti code that supports your implementation, you really need to make sure that new features have not completely blown you out of the water. I have seen something as small as lengthening a textbox totally destroy a UI, and I have seen something as simple as the user entering a comma instead of a decimal point make mincemeat out of data.
The high level functionality was easily written into “Sunny Day” User Story scenarios. “Blue Sky, Smiling Sunshine” scenarios are where the user does what they want to do and encounters no surprises, the software behaves perfectly, and the user has a flawless experience. These were simple to articulate.
After writing these up in straightfoward narrative, we went through and tested these User Stories by following along with the Sunny Day Scenario. A few bugs showed up. Thankfully, through disciplined Bugzilla tracking and a talented development team, we squished these with impunity.
(Bugzilla = Open Source and Open Source makes me happy.)
After tackling the obvious, we tested the same functionality in different browsers. It is surprising how many things break in IE7 or FireFox. Part of me can’t understand how this is possible if we have standards and develop in adherence with them. The other part of me knows better.
From there, we pretended to take off our development hats and forget everything we knew about the software. You can pretend to be uninformed or completely incompetent. You can go through and click the wrong links, leave mandatory fields empty, do any number of things that should be handled by validation routines or error handling. You can pretend.
Bring people in who have nothing to do with your project and have them attempt to act out the key User Stories. See what happens. Make note of what they do and especially what they do that you didn’t expect them to do. Document everything as you go along - not just after you feel you have completed a test case. Document what you do as you are doing it. Otherwise, it may be very difficult to reproduce something. Remember, you do not have a Test Case. You are flying solo, in unchartered… er… skies?
Be sure to click every link and press every button, even when they are not relevant to the task at hand. They make software that does this kind of mundane task, but I did not have access to it at the time and you may not. You can choose to outsource this gruntwork to QA houses. They will be happy to charge you $120 an hour. It is your choice if you want to engage them. I don’t tend to like that approach. Throwing money at QA is fine, but not when it is hurried and ambiguous QA. Without a formal testing plan to map to, there is almost no accountability for the QA consultant. Consultants need to be able to be held accountable if they are to be a wise investment.
I recommend making a a simple .xls that you work off of as you go along, filling in a “test case” number, a descriptive title, a field for the initials of the brave tester, a field that describes expected System behavior, a pass/fail field, and a space for notes. This may wind up being valuable. It may obviate that there is a general issue with a specific type of validation, or that a URL keeps mapping back to a hardcoded URL that points to your staging server, or otherwise clarify issues that would otherwise exist only as anecdotes.
Finally, bang away, trying to break it. Put erroneous values in deliberately. Hit “back” when you are not supposed to. Be crazy. Wear a funny hat. Try to make the software choke, cough, or worse.
There is no such thing as testing without a testing plan, but for us, this approach has worked pretty well - and yielded some rudimentary documentation we were able to point to afterwards. Start with the happy scenarios, move to the exceptions, then shimmy to the downright silly, ugly, and hacky.
It isn’t the ideal way to do things, but it is a heck of a lot better than just assuming it works.
I certainly don’t recommend this approach, and would probably deny that I wrote it, but sometimes you have to work with what you have and do the best you can despite the circumstances.
