Microsoft and automated testing
Posted by Pierre Igot in: Microsoft, TechnologyJanuary 21st, 2005 • 2:19 am
This little piece of news at C|Net might not sound like much, but I find it significant, because it echoes stuff I have heard from other sources about Microsoft moving more and more towards automated testing instead of using actual human beings to test their software.
Why is it significant? Because Microsoft’s software is already bad enough as it is when it comes to the way it interacts with the end user as a live human being. Automated testing might help identify certain kinds of bugs — but I am also pretty sure that there are all kinds of real-life scenarios that automated testing will never be able to replicate — not least because there’s no reason to think the folks who design the automated tests are any less clueless about how real people use software in real life situations than the people who design the software itself.
Here’s an example of automated testing that I did once, in order to try and identify the scenarios that would lead to the recurrence of the dreaded “Disk Is Full” bug in Microsoft Word. The automated test was a complete failure, meaning that it was utterly unable to reproduce the bug, even though I really thought that it would.
After many years of suffering, I finally managed to get through to someone at Microsoft about this particular bug (albeit at the cost of the pages upon pages of rather ineffective “conversation” that ensued), and it looks as if my bug report did finally help identify at least some of the scenarios that would lead to the recurrence of the bug. At least that’s what I gather from the fact that the Microsoft engineer in question did request some additional information from me at the time (including a sample file) and that I haven’t encountered the bug a single time since upgrading to Word 2004.
My point here is simply that it takes more than a battery of automated tests to identify bugs and flaws in software programs. Most software applications rely on a huge amount of interaction with live human beings, and many of the problems with these applications are intrinsically linked to this interaction — and, last time I checked, live human beings were not automated.