Static Vs Dynamic: Foundations Of Testing

Hey folks, today I wanted to kick off a series that compares and contrasts dynamic testing with static code analysis, and digs into how the two intersect to form a holistic testing strategy.

First off, let’s start with some definitions and a general guide to what each of these is used for. Dynamic testing is the practice of loading up some or all of the application code in a runtime environment and verifying how it executes. This can be further broken down into functional and non-functional testing. Functional testing verifies that the code does what it’s supposed to do. This can be very narrow, such as a single function being executed in a unit test, or very broad, such as a Docker image being deployed to a staging environment and exercised by Cypress. Non-functional testing verifies other aspects of an application’s behavior. A classic example is performance testing which verifies how quickly a web application responds to a request or renders a page.

Static code analysis on the other hand inspects the source code on the file systems to look for areas of concern. This can be for superficial purposes like applying a standard coding style guide for indentation and line breaks. More importantly though, this can alert us to bugs in the code, such as identifying when we’re using a function that doesn’t exist in older versions of a browser. Compiled languages get a sort of static code analysis for free, as their compiler will error out if certain issues exist in the code. Languages that are a little more loosey-goosey rely on an ecosystem of tools like ESLint and TypeScript to add additional static checks to the build pipeline.

While there is some overlap in what the two can do, one is typically better suited for a particular job than the other. Take type safety as an example. You could invoke a function with a bunch of different inputs and see if it ever fails due to a type mismatch. However, as we know from the first of the ISTQB Seven Testing Principles, “Testing shows the presence of defects, not their absence.” We can say that we never observed a type mismatch in our testing, but we can’t say for certain using this methodology that the caller will never have a type mismatch. Static code analysis much more easily verifies what is disallowed by comparing an expected type against the list of allowed types specified in the code. This also highlights a fundamental difference between black-box testing, which ignores implementation details, and white-box testing, which relies on a deeper knowledge of the code.

The list of specialized use cases goes on. Dynamic testing can identify issues in the behavior of the code even when the code itself is syntactically correct. Think logical bugs like off-by-one errors. Dynamic testing can also look for problems beyond the current codebase, like performance bottlenecks related to our infrastructure. Meanwhile, static code analysis can surface more nuanced maintainability issues that wouldn’t be visible at runtime like high cyclomatic complexity in the code.

I imagine this is a lot of review for more experienced developers. I wanted to walk through this comparison though to level-set ahead of the investigation I find more interesting: how the two can be used in conjunction to tackle complex testing issues. We’ll dig into that more in the next part of this series!