The Dark Side of Test Automation
Industry-Wide Problems
As I reflect on my experiences working with automation testers, I’ve come to realize that we’re pouring vast sums of money, time, and resources into a faulty system. It’s not about pointing fingers; it’s about acknowledging that our approach to test automation is flawed.
The Rise of the Automation Tester
We’ve created a new breed of developer – the automation tester – whose primary goal is not only to find bugs but also to write regression tests to avoid re-running manual tests. Sounds great in theory, but in reality, these tests are often brittle and prone to failure.
The False Promise of End-to-End Tests
End-to-end tests, typically written in Selenium WebDriver, sound impressive, but they’re often deleted due to their non-deterministic nature. These tests sabotage our CI builds, making progress nearly impossible. We’re too busy writing and maintaining these underperforming tests to bother with manual testing.
The Test Pyramid: Forgotten but Not Lost
We’ve forgotten the test pyramid, where unit tests and integration tests carry the bulk of the testing burden. Instead, we’re relying too heavily on end-to-end tests, which are expensive to maintain and often yield little value.
Selenium WebDriver: Not Fit for Purpose
Selenium WebDriver is not suitable for dynamic web pages, leading to flaky tests and maintenance nightmares. Cypress.io offers a better solution by embedding itself in the browser and executing tests synchronously.
The Effectiveness of Tests: Untracked and Unmonitored
We’re not tracking the effectiveness of our tests, and we’re not deleting bad tests. We need tooling that monitors test flakiness and quarantines tests that fail repeatedly.
Non-Deterministic Tests: The Enemy of Progress
Rerunning tests on failure is a sign of defeat. We must eradicate non-deterministic tests and focus on writing reliable, deterministic tests.
The High Price of Maintenance
Maintaining end-to-end tests comes at a high cost. When it takes more effort to update tests than to re-run them manually, test automation initiatives fail. We must treat automation test code as production code, avoiding copy-and-paste hacks and ensuring tests are independent and repeatable.
Testers No Longer Want to Test
Manual testing is perceived as outdated and boring, leading many automation testers to abandon it altogether. We must remember that manual testing catches many more bugs than writing one test with one fixed set of inputs.
Questions That Need Answers
Why are we pouring resources into something that’s not working? Why do we accept flaky tests as the norm? Why do we re-run tests with the same inputs and expect different results? Why do we still use Selenium WebDriver despite its limitations?