Manual testing vs automated testing

Manual testing — where a real live person clicks around and verifies that the code does everything it's supposed to do — has its uses. But for large-scale projects, or in cases where you need to test the same code repeatedly, automated testing can be both more cost-effective and more fruitful. You know how your eye can look at the same spelling error six times without seeing it? Automated testing can catch issues like that. It's great for rote, boring tasks that humans gloss over.

In test-driven development, you actually build the tests first and then write the code that fulfills those tests. But you don't necessarily have to do automated testing that way. Tests can be written afterwards. Sometimes, it's old code that gets automated tests built for it.

Writing the test probably takes more time than initially testing it. But then it's done, and you can re-run the test any time you make any change to that site. Even if you don't change anything near that code, you can run the test anyway and catch those instances where you're like, "I'm sure it won't break that"…but it does.

Drupal maintainers try to be really strict about automated tests. Since lots of people use the modules and contribute to them, you have all these different people submitting code. Having a good test suite can really improve your confidence in a module. If each submission comes with new tests that you can run and verify, you can be far more confident in the quality of that code.

Time savings

Say you do 10 hours of manual testing for each release. If, on the other hand, you spend 10 hours writing automated tests for each release, then for release #2, you're actually doing 20 hours of testing, and for release #3, you're doing 30 hours of testing. You're only putting in 10 man-hours each time, but your test suite keeps growing, and the scope of what you're testing increases exponentially because you can rerun the same tests each time.

Why is automated testing such a hard sell?

Going back to the flossing example: Why don't you floss? It takes 60 seconds. But you only really get the benefits if you do it all the time.

More to the point: Not doing it takes time to become bad. Skipping automated testing on your first release may not be a big deal. Your codebase is small; it's probably only doing a couple of things, so manual testing is very feasible. But as the project grows and gets more complex, manual testing becomes increasingly unwieldy, and then you get to a point where you think, we're too far into this to add automated testing now.

But the truth is you can start at any point. It's never too late to start proactively doing things that will benefit your site.

What prevents people from getting started?

Usually, the thing that stymies people is that they're not set up for automated testing: They don't have a continuous integration environment or a nice testing environment to run the tests on. Or maybe they've neglected it for so long that the regular tests don't work anymore; they write their first test and they have all these other problems because they've let things languish, so they give up.

What is continuous integration, you ask? It means every time you do a change and you push it out, it attempts to integrate it with the whole project. It will deploy it to a server, it will compile the code if necessary, it will run code standards and automated tests and so on. When you have that stuff all set up to run automatically, you just make the code, push it to your version control, and forget about it. If something goes south, it'll send you an email saying this didn't pass. Otherwise, you don't have to think about it again.

How much of your code should be covered by automated testing?

You kind of want to be in that 95% range, although it's true that you can have parts that aren't easily testable. You can cover a lot of code, but you might still be missing use cases. Maybe you test taxes, and you test discounts, but you don't test taxes and discounts together. So technically you have full code coverage, but you're not using them in combination. So code coverage is a nice metric, but it doesn't tell the whole story.

TL;DR: Automated testing is like flossing. You should really do it.

More from Acro Commerce

Drupal Commerce 2 Technical Overview for Developers and Technical Managers
Tech Talk video: Drupal Website Debugging and Site Performance
Learn more about Acro Commerce

Chat with us

If you'd like to talk about adding automated testing to your Drupal Commerce website, or if you just want to discuss how Drupal Commerce fits into your ecommerce solution, give us a shout. We're happy to chat.