Automate Tests
Automate tests for your codebase. Do this by writing automated tests using a test framework. This improves maintainability because automated testing makes development predictable and less risky.
Automated unit tests are tests of code units themselves described in code that runs autonomously. The same holds for other types of testing, such as regression tests and user acceptance tests: automate as much as possible, using a standard test framework. For unit tests, a common framework is jUnit.
Motivation
This section describes the advantages of automating your tests as much as possible.
Automated Testing Makes Testing Repeatable
Just like other programs and scripts, automated tests are executed in exactly the same way every time they are run. This makes testing repeatable: if a certain test executes at two different points in time yet gives different answers, it cannot be that the test execution itself was faulty. One can conclude that something has changed in the system that has caused the different outcome. With manual tests, there is always the possibility that tests are not performed consistently or that human errors are made.
Automated Testing Makes Development Efficient
Automated tests can be executed with much less effort than manual tests. The effort they require is negligible and can be repeated as often as you see fit. They are also faster than manual code reviews. You should also test as early in the development process as possible, to limit the effort it takes to fix problems.
Important: Postponing testing to a late stage in the development pipeline risks late identification of problems. That costs more effort to fix, because code needs to go back through the development pipeline and be merged, and tests must be rerun.
Automated Testing Makes Code Predictable
Technical tests can be automated to a high degree. Take unit tests and integration tests: they test the technical inner workings of code and the cohesion/integration of that code. Without being sure of the inner workings of your system, you might get the right results by accident.
A common advantage of automated testing is identifying when regression is occurring. Without a batch of automated unit tests, development quickly turns into a game of whack-a-mole: you implement a change in one piece of code, and while you are working on the next change in another piece of code, you realize you have introduced a bug with your previous change. Automated tests allow you to double-check your entire codebase effortlessly before turning to the next change. And since the automated unit tests follow predefined paths, you can be sure that if you have fixed a bug, it does not pop up on a second run.
Thus, running automated tests provides certainty about how the code works. Therefore, the predictability of automated tests also makes the quality of developed code more predictable.
Tests Document the Code That Is Tested
The script or program code of a test contains assertions about the expected behavior of the system under test.
The assert statement plays a double role: as the actual test, and as documentation of the expected behavior. In other words, tests are examples of what the system does.
Writing Tests Make You Write Better Code
Writing tests helps you to write testable code. As a side effect, this leads to code consisting of units that are shorter, simpler, have fewer parameters, and are more loosely coupled (as the guidelines in the previous chapters advise).
For example, a method is more difficult to test when it performs multiple functions instead of only one. To make it easier to test, you move responsibilities to different methods, improving the maintainability of the whole. That is why some development approaches advocate writing a unit test before writing the code that conforms to the test. Such approaches are called test-driven development (TDD) approaches. You will see that designing a method becomes easier when you think about how you are going to test it: what are the valid arguments of the method, and what should the method return as a result?
How to Apply the Guideline
How you automate tests differs by the types of tests you want to automate. Test types differ in what is tested, by whom, and why. They are ordered from top to bottom based on the scope of the tests. For example, a unit test has the unit as scope, while an end-to-end test, a regression test, and an acceptance test are on the system level.
Unit test
Functionality of one unit in isolation
Verify that unit behaves as expected
Developer (preferably of the unit)
Integration test
Functionality, performance, or other quality characteristic of at least two classes
Verify that parts of the system work together
Developer
End-to-end test
System interaction (with a user or another system)
Verify that system behaves as expected
Developer
Regression test
Previously erroneous behavior of a unit, class, or system interaction
Ensure that bugs do not re-appear. Verify that all unchanged parts of a system still function correctly after the implementation of a change.
Developer
Progression test
Test of new or adapted parts of a system.
Verify that new or parts of the system behaves as expected
Developer
Acceptance test
System interaction (with a user or another system)
Confirm the system behaves as required
End-user representative (never the developer)
Different types of testing call for different automation frameworks.
For unit testing, several well-known Java frameworks are available, such as jUnit.
For automated end-to-end testing, you need a framework that can mimic user input and capture output. A well-known framework that does just that for web development is Selenium.
For integration testing, it all depends on the environment in which you are working and the quality characteristics you are testing. SoapUI is a framework for integration tests that focuses on web services and messaging middleware.
Apache jMeter and Gatling are frameworks for testing the performance of Java applications under heavy workloads.
Choosing a test framework needs to be done at the team level. Writing integration tests is a specialized skill—but unit testing is for each and every individual developer. Java developers are writing unit tests using the most well-known framework jUnit.
General Principles for Writing Good Unit Tests
When writing tests, it is important to keep in mind the following general principles:
Test both normal and special cases
Test two kinds of cases:
Write tests that confirm that a unit indeed behaves as expected on normal input (called happy flow or sunny-side testing).
Also, write tests that confirm that a unit behaves sensibly on non-normal input and circumstances (called unhappy flow or rainy-side testing). For instance, in jUnit, it is possible to write tests to confirm that a method under test indeed throws a certain exception.
Maintain tests just like nontest (production) code
When you adjust the code in the system, the changes should be reflected in the unit tests as well. This is most relevant for unit tests, though it applies to all tests. In particular, when adding new methods or enhancing the behavior of existing methods, be sure to add new test cases that cover that new code.
Write tests that are isolated: their outcomes should reflect only the behavior of the subject being tested
That is, each test should act independently of all other tests. For unit testing, this means that each test case should test only one functionality. No unit test should depend on state, such as files written by other tests. That is why a unit test that, say, causes the class under test to access the filesystem or a database server is not a good unit test.
Measure Coverage to Determine Whether There Are Enough Tests
How many unit tests are needed? One way to assess whether you have written enough unit tests is to measure coverage of your unit tests. Coverage, or more precisely, line coverage, is the percentage of lines of code in your codebase that actually get executed when all unit tests are executed. As a rule of thumb, you should aim for at least 80% line coverage with a sufficient number of tests—that is, as many lines of test code as production code.
Why 80% coverage (and not 100%)? Any codebase contains fragments of trivial code that technically can be tested, but are so trivial that testing them makes little sense. Take the following typical Java getter method:
A minimum of 80% coverage alone is not enough to ensure high-quality unit tests. It is possible to get high coverage by testing just a few high-level methods and not mocking out lower-level methods. That is why we advise a 1-to-1 ratio of production code versus test code.
You can measure coverage using a code coverage tool. Well-known examples are the Maven/Gradle plugin Jacoco, and the Eclipse plugin EclEmma.
Examples:
Common Objections to Automating Tests
This section discusses typical objections and limitations regarding automation. They deal with the reasons and considerations to invest in test automation.
Objection: We Still Need Manual Testing
“Why should we invest in automated tests at all if we still need manual testing?”
The answer to this question is simply because test automation frees up time to manually test those things that cannot be automated.
Consider the downsides of the alternative to automated tests. Manual testing has clear limitations. It is slow, expensive, and hard to repeat in a consistent manner. In fact, technical verification of the system needs to take place anyway, since you cannot manually test code that does not work. Because manual tests are not easily repeatable, even with small code changes a full retest may be needed to be sure that the system works as intended.
Objection: I Am Not Allowed to Write Unit Tests
“I am not allowed to write unit tests because they lower productivity according to my manager.”
Writing unit tests during development actually improves productivity. It improves system code by shifting the focus from “what code should do” toward “what it should not do.” If you never take into account how the code may fail, you cannot be sure whether your code is resilient to unexpected situations.
The disadvantages of not having unit tests are mainly in uncertainty and rework. Every time a piece of code is changed, it requires painstaking review to verify whether the code does what it is supposed to do.
Objection: Why Should We Invest in Unit Tests When the Current Coverage Is Low?
“The current unit test coverage of my system is very low. Why should I invest time now in writing unit tests?”
We have elaborated on the reasons why unit tests are useful and help you develop code that works predictably. However, when a very large system has little to no unit test code, this may be a burden. After all, it would be a significant investment to start writing unit tests from scratch for an existing system because you would need to analyze all units again. Therefore, you should make a significant investment in unit tests only if the added certainty is worth the effort. This especially applies to critical, central functionality and when there is reason to believe that units are behaving in an unintended manner. Otherwise, add unit tests incrementally each time you change existing code or add new code.
Last updated
