Monday, March 27, 2017

The Myth of Automating without Exploring

I feel the need of calling out a mystical creature: a thinking tester who does not think. This creature is born because of *automation*. That somehow, because of the magic of automation, the smart, thinking tester dumbs down and forgets all other activities around and just writes mindless code.

This is what I feel I see when I see comparisons of what automation does to testing, most recently this one: Implication of Emphasis on Test Automation in CI.

To create test automation, one must explore. One must figure out what it is that we're automating, and how could we consistently check the same things again and again. And while one seeks for information for the purposes of automation, one tends to see problems in the design. Automation creation forces out focus in detail, and this focus in detail that comes naturally with automation sometimes needs a specific mechanism when freeform exploring. Or, the mechanism is the automation thinking mindset. 

I remember reading various experience reports of people explaining how all the problems their automation ever found were found while creating the automation. I've had that experience in various situations. I've missed bugs for choosing not to automate because the ways I chose to test drove my focus of detail to different areas or concerns. I've found bugs that leave my automated tests in "expected fail" state until things get fixed.

The discussion around automation is feeling weird. It's so black and white, so inhumane. Yet, at core of any great testing, automated or not, there is a smart person. It's the skills of that person that turn the activity into useful results. 

Only the worst of the automators I've met dismiss the bugs they find while building the automation. Saves them time, surely, but misses a relevant part of feedback they could be providing. 


  1. Maaret,
    I agree wholeheartedly with you on this. If you are not exploring and testing the behavior of the SUT as you begin (and continue as you work with the system) the automation work you are not doing your job. Blindly writing code is not going to cut it. Pure "Happy Path" testing/navigation of the system will only go so far. In order to understand how the system really works, and how best to have the automation interact with it, means doing your own "self testing". You have to get your own hands dirty in the functionality to see its behavior. Once you've learned how the system behaves then you can get the automation to act accordingly. Automation is only as good as the person who built it, and part of that is the front-end work of the person building it.

    Jim Hazen

  2. I completely agree. Much of what I see on test automation focuses entirely on the implementation or technology side of it (in part probably because vendors are out there selling), and seems to ignore the fact that creating good test automation inherently requires the same skills that a "manual" tester needs.

    One unfortunate reality is there are automation engineers out there who do not create or design tests, but simply take fully-specified test cases and implement them. This is such an inefficient waste of peoples' time (and runs into some of the same problems that Agile and XP exist to solve), I can't begin to describe how much I hate it. But it's real.

  3. Completely agree ... this dichotomy that is being propagated is creating an artificial chasm between the "exploratory" and "automated" practices of testing . Anyone who claims to have exercised both components of the Testing craft will tell you that both crafts have their roots in incisive critical analysis,focus and skill . In fact they are complimentary set of skills and mindset ... a contemporary tester can not be effective without either.