Is Automated Software Testing Enough?

Automated testing is all the rave lately, but there is so much more around automated testing that we shouldn’t forget.  I was recently listening to a Scott Hanselmen talk with Queztal Bradley about testing.  Queztal is a software development engineer on Chris Sell’s team with a crazy amount of Real World Software Development experience, focusing around software testing.  They had a lot of fantastic points during their discussion, but if you don’t have time to listen to it, let me summarize some of the highlights.

Code Coverage is an Inverse Number

Let’s say you have 50% code coverage within your application.  In reality you have very little code coverage.  Why?  Well, you have about half of your application completely untested.  That is a scary scenario.  A nice analogy (I love analogies) is to look at it as if you live on 8 Mile in Detroit.  You have just installed locks on half of your doors in your house.  Does this really help you at all? 

I mean sure you should feel pretty good about yourself for installing the locks on half the entry points, but locks on only half of your doors and windows is not going to help you very much.  The burglar will just try everything anyway and get into your palace of software creation.  Even if you have locks on everything, can they still get in?  Absolutely! Just break a window.  And that’s where we get into the rest of the problems with just having 100% code coverage.

Sample Testing with Diverse Data

One of the great concepts he talks about is getting your application to “phone home” to get data to put it into your unit tests.  I’ve always thought unit tests were great, but the problem is, how many variations of data are you running through your application.  How many times have I seen a software developer write a “wicked” unit test, that only really test a few variations of data.  Two problems here.  Not enough data to run the test with, and the second problem, is the data actually real world?  There are a bunch of ways you can get real world data from your users of course.

One great way to do this is to tie up a web service to your application to send data whenever exceptions occur within the application.  Do you remember Dr. Watson?  And no I’m not talking about Sherlock Holmes friend and confidante.  I’m talking about good old drwatson.exe!That application was probably one of the most used and important applications Microsoft ever created, and still uses today.

The other method of course is to phone up your customers and get real world data from them.  Most times they are more than happy to provide it to you.  You really need this data!  It’s simply not enough to write a unit test that tests a section of the application for one or two expected outputs, and only uses a few variations of the inputs.  This simply does not simulate a real world test, and really gives you a false sense of security.

State Coverage is Important

State coverage?  How many different “states” will your customer place your application in.  Are you testing for state coverage? State coverage measures whether unit-level tests check the outputs and side effects of the program.  State coverage is a whole other article which I will cover in the coming weeks, but you should definitely start looking into.  For example, state coverage will tell you exactly how many assert-less unit tests your software development buddies have written just to make sure they have code coverage.

Feedback and AdHoc Manual Testing

At the end of the day, all the automated testing in the world is not going to catch all of the issues with your application.  Let’s say you test your application, all seems well, but for some reason a button has turned purple.  Your automated tests won’t catch this.  Should you go and write a test to test this?  Write a test to test this for all buttons? Drop Downs?  This is a simple example of something that can be caught very easily by humans, but not so easily by computers.  While you can take your automated testing to a crazy level, sometimes you should stop and think if its really necessary.  There is always a place for Manual Testing.

When you receive feedback that a bug is in the software, or a manual test has been run by a software tester and a problem found, what do you do?  Probably 90% of you just go in and fix the problem.  Is this a good approach?  Not really.  What the feedback or manual testing has done is uncovered an issue with your current automated tests.  This is the absolute best time for you to go into the application, and write some unit tests.  Regression found? Excellent, write a unit test to re-create the regression, then fix the regression.  Run your tests. Is the regression fixed?

Anyway, you can listen to the show in its entirety at Hanselminutes.com

If you liked this article, please share it on DZone, del.icio.us, StumbleUpon or Digg. I’d appreciate it.