Skip to main content
U.S. flag

An official website of the United States government

Testing and Automation

By Software Management

As a software developer, testing your code to make sure it works is a given. Even the simplest scripts require some level of testing to ensure that a prescribed set of inputs results in the expected outputs. When code or software are distributed without thorough testing, often a lengthy period of fixing errors, bugs, and other problems follows.

Testing is applicable throughout the lifespan of a software. If the software is widely used or achieves some level of mission criticality, maintenance might extend out over years resulting in the need to address operating system upgrades, deprecations and upgrades to underlying technology stacks, and staff turnover resulting in the need to train new people how the software works. During all of this, the developer still needs to test the code to ensure it is of high quality and does what it is supposed to do despite myriad incremental changes applied over time. Additionally, experienced developers who use open-source tools will often look at the tests to better understand Application Programming Interfaces (API) and tool functions. Well-written tests can serve as developer documentation for a component’s interfaces.

Consider writing tests that can be automated. There is a large body of evidence that attests to the efficacy of a practice called “Test Driven Development” in software development. You are never as familiar with the code as when you are implementing it, so it is very helpful to simultaneously write tests while you are writing the implementation. Having a set of tests, often called “unit tests” that prove a function/process/method works as expected, becomes the source for tests that can be run automatically.

Why Automate?

The following sections provide information on automation tools and techniques that might help to improve the quality, efficiency, and maintainability of your work. Automating tasks that improve the quality of your software is a vast topic and there are hundreds of choices a developer can make with respect to tooling and design. Find an approach that best fits your project, team, and coding style.

Automated Build

A software project's build is a process that converts source code into a consumable product. During a typical build, a software project's dependencies are retrieved, its source code is compiled, and the resulting executable code is packaged for distribution. A build should fail if any of these steps fails. A build should be run early and often throughout the development process to verify the quality of the software.

Automating the build ensures that it is easy to run the build often, and makes it easy for new programmers to contribute to a software project. The ideal automated build should enable someone on a generic machine to retrieve the source code from a version control repository, issue a single command, and successfully produce a package of executable code suitable for distribution. To reduce the setup burden on other programmers, or on an automated build server, a build should be able to complete successfully without using an Integrated Development Environment (IDE) or other graphical user interface.

Many tools exist that support automated builds. A non-exhaustive list follows.

Automated Testing

It is useful  to run automated tests during the build process. The build may prove that the software can be built without compilation errors, however, automated tests show that the software performs its intended function. It is widely accepted throughout the software development industry that automated tests, rather than manual tests, are essential for consistently delivering cost-effective software that meets users' evolving needs in a timely manner with minimal defects. Spending a small amount of time automating testing saves time down the road. Automated tests reduce the incidence of defects. Writing automated tests throughout the life of the project also encourages code to be better designed. The cost of creating, maintaining and running automated tests is counterbalanced by the ability of programmers to innovate more with smaller risk of breaking existing functionality.

Pete Goodliffe expands on this idea in Becoming a Better Programmer:

"If in the future someone (even you on a bad day) modifies the code to introduce errant behavior (a functional regression), the test will point this out immediately … Yes, writing these tests takes up the programmer's precious time. And yes, your confidence in the code is only as good as the quality of the tests that you write. But it's not hard to adopt a test strategy that improves the quality of your code and makes it safer to write. This helps reduce the time it takes you to develop code ..."

Unit Testing

There are many varieties of automated testing. A common variety is unit testing.

Definition

"A short program fragment written and maintained by the developers ... which exercises some narrow part of the product's source code and checks the results. The outcome of a unit test is binary: either "pass" if the program's behavior is consistent with the recorded expectations, or "fail" otherwise." - Agile Alliance, "Unit Testing"

Unit testing is usually done in an automated fashion as a part of the project's build so that program correctness is verified early and often throughout development. To support this goal, all of a project's unit tests should be able to execute within a few seconds. Unit tests should also be able to be run on any programmer’s or build server's environment. For the sake of speed and reproducibility, unit tests rarely rely on resources that are outside of the code's version control repository (e.g. large binary files or volatile network resources). Writing your code such that unit tests can be performed provides technical reviewers and yourself with a thorough and granular method to review the functionality of the code at any time.

There are many free unit testing tools. A non-exhaustive list follows.

Other Types of Automated Testing

There are many other types of automated testing including Integration Testing, End-to-End Testing, and Performance Testing.

Informational resources:

Automated Code Quality Metrics

In addition to automated testing, software projects are increasingly adopting automated code quality metrics. Code quality metrics analyze programs' source codes and runtime behaviors to discover areas for improvement in correctness, design, security, performance and other concerns. Incorporating code quality metrics into the automated build ensures that programmers get feedback about their code quickly and more often.

There are two broad categories of metrics: static analysis and dynamic analysis. Static analysis is accomplished by inspecting the source code without running any code. Dynamic analysis requires running the code.

Static Analysis

Some static analysis metrics are simple. For example, static analysis tools might raise alarms if functions have too many lines. This encourages the creation of smaller functions, which are generally easier to comprehend, test, and maintain. Other metrics are more involved. For example, the cyclomatic complexity metric measures the number of independent paths through a function. The more levels of nested control flow (loops, conditions, etc.) there are inside of a function, the higher the cyclomatic complexity. In general, the higher the cyclomatic complexity, the more difficult the code is to comprehend.

Dependency analysis is important too. It inspects the versions of a project's dependencies, compares them against lists of known security vulnerabilities, and against the most up-to-date versions of the dependencies. Automated dependency analysis helps programmers plan efforts to upgrade dependencies and reduces response time to security vulnerabilities.

There are many static analysis tools. A non-exhaustive list follows.

Wikipedia has a more comprehensive index.

Dynamic Analysis

Test Coverage

A common dynamic analysis metric is test coverage (also known as code coverage). Test coverage is the percentage of code that is executed by automated tests. A codebase's overall test coverage provides an indication of how well-tested the codebase is. Line-by-line test coverage data can highlight lines of code that are not being tested. A programmer can use test coverage data to improve tests, and to better-inform estimates of how long a given task will take.

Informational resources:

There are many test coverage tools. A non-exhaustive list follows.