Test Coverage with Pytest
Code coverage is a measure of what percentage of lines of code are covered by a test, identifying the unused conditional branches and lines.
Gaps in testing can be identified and assessed by running a utility, such as Python’s coverage utility. These can be assessed and either ignored, perhaps for being trivial, or tests written to increase coverage. A good target for test coverage is eighty percent.
Why would you want to write that many tests, it seems like a lot of hard work?
Consider an application that you have to support but know little about. Perhaps the original team of developers no longer work in the company. You are asked to make a change.
If there is a decent amount of test coverage, you can run the tests after you make a change to verify that your change did not affect any functionality. So long as the functionality is properly tested, that is.
Not all tests are written equally. A good coverage score does not necessarily mean that code is safe to work on.
I have inherited code with tests that have no assertion statements. The tests ran every time the module was built, but proved nothing. If an expectation is not checked then how do you know the code is working?
What’s worse is that these tests were inflating the coverage number given in our Sonar build making it look like the code had good test coverage. While there may be some value to exercising the code, it would be a lot more helpful to include assertions.
At the minimum, assertions help act as documentation to what the code is trying to do. When written well, they can stop stupid mistakes made by a developer changing something without understanding the impact.
Adding Coverage
In this short post I am going to use coverage to measure the existing test coverage that I have on the Anonymous Agile Estimator application I have been slowly building.
The first thing is to install the tool. After activating the virtual environment that I have been using, this command will install the necessary code.
pip install coverage
After installing something new into the virtual environment, it is a good idea to update the requirements.txt file so that the project can be rebuilt. I only need to run the test coverage in development, so they are not required in the prod_requirements.txt file.
pip freeze >requirements.txt
Running the Tests
Running the tests is straightforward. If you run it without specifying the source modules to monitor it will include all the dependent modules. We don’t want all that, so we will specify to analyse the estimator and database modules only using the --source option.
coverage run --source estimator,database -m pytest
The usual Pytest output will be displayed. The utility saves the details into a hidden file called .coverage which should be added to the .gitignore, if not already there.
To see the output from the coverage run the following, the option is to show all line numbers that have been missed.
Name Stmts Miss Cover Missing
------------------------------------------------------------
database/__init__.py 0 0 100%
database/models.py 8 1 88% 13
estimator/__init__.py 17 0 100%
estimator/controllers.py 7 0 100%
estimator/rest_controller.py 23 0 100%
------------------------------------------------------------
TOTAL 55 1 98%
So the application is missing a test for line 13 of the models.py file, which is in the __repr__ function for the Group class as follows:
def __repr__(self):
return '<Estimation group: {}>'.format(self.name)
I don’t think that there is any need to add a test for this functionality. It could be argued that this serves only an internal debugging requirement. However, since I like completeness I will add the following test to get the statistics up to one hundred per cent (I swear I am not OCD, at least not all the time).
from database.models import Group
def test_repr():
expected = '<Estimation group: GroupName>'
group = Group('GroupName')
assert expected == group.__repr__()
Now my code is fully covered by tests, each of which have assertions.
$ coverage report -m
Name Stmts Miss Cover Missing
------------------------------------------------------------
database/__init__.py 0 0 100%
database/models.py 8 0 100%
estimator/__init__.py 17 0 100%
estimator/controllers.py 7 0 100%
estimator/rest_controller.py 23 0 100%
------------------------------------------------------------
TOTAL 55 0 100%
All is right with the world. Are we done?
Branch Coverage
Not quite. The above commands only measured line coverage. What if a line contained an in-line conditional. The line might be used in a test, but have both sides of the conditional been evaluated?
To run the coverage with branch analysis a new argument is required, which is false by default. Here’s what happens when I ran it with these settings on:
$ coverage run --source estimator,database --branch -m pytest
========================================================================== test session starts ==========================================================================
platform darwin -- Python 3.6.1, pytest-3.8.2, py-1.7.0, pluggy-0.7.1
rootdir: /Users/rkilleen/projects/python/agile-estimation/estimator, inifile:
collected 9 items
tests/test_controller.py .... [ 44%]
tests/test_models.py . [ 55%]
tests/test_rest_controller.py .... [100%]
======================================================================= 9 passed in 0.39 seconds ========================================================================
$ coverage report -m
Name Stmts Miss Branch BrPart Cover Missing
--------------------------------------------------------------------------
database/__init__.py 0 0 0 0 100%
database/models.py 8 0 0 0 100%
estimator/__init__.py 17 0 0 0 100%
estimator/controllers.py 7 0 0 0 100%
estimator/rest_controller.py 23 0 4 0 100%
--------------------------------------------------------------------------
TOTAL 55 0 4 0 100%
No additional problems specified, but the code is quite simple so far and this is not unexpected.
Having to remember these switches and arguments everytime I run the coverage analysis is going to be annoying. Is there a way to save these settings?
Yes. I am going create a .coveragerc file to store my default settings. It can store settings for all of the sub commands coverage has, but for now all I want to include are the run and report settings.
Note that coverage will also read settings from several other locations, for example, setup.cfg if it exists, however these need to be prefixed with the name of the tool, changing [run] to [coverage:run].
[run]
branch=True
source =
estimator
database
[report]
show_missing=True
Now I can generate the test coverage with coverage run -m pytest and show the report with coverage report.