Skip to content

Testing#

Tooling#

Configuration#

Both pytest and coverage.py support configuration via the pyproject.toml file. Learn more at their respective reference documentation.

Writing Tests#

Python tests are implemented using pytest in the tests subdirectory.

docs/examples/default/tests
├── test_cli.py
└── test_simple.py

Each test module starts with test_ so it is automatically discovered when running pytest.

In addition, pytest also collects data from a conftest.py file. This would be where global fixtures are defined that can be reused across multiple test modules.

Executing tests#

...during development#

Tests can be executed like this

$ pytest -q
......                                                                   [100%]
6 passed in 0.03s

If you have never used pytest before, check out their pytest documentation. The more you know about pytest, the better your test suite is going to be. 😉

...as CI job#

If a remote has been configured, your tests are also automatically run as part of the project's continuous integration pipeline.

Run tests in GitLab CI 
test:
  cache: # reuse venv in subsequent jobs
    key: $CI_JOB_NAME
    paths:
      - .cache/pip
      - env/
  before_script:
    - python -m venv env
    - source env/bin/activate
  script:
    - pip install .[test]
    - pytest --doctest-modules --cov --cov-config=pyproject.toml --cov-branch --cov-report term --cov-report html:build/coverage --junitxml=report.xml --cov-report xml
    - pip install anybadge==1.9.0
    - mkdir -p build/badges
    - pip install radon==5.1.0
    - make maintainability
    # generate a badge for the maintainability index with the total average of cyclomatic complexity as value
    - |
      score=$(python -m radon cc --total-average src | tail -n 1 | cut -d' ' -f 3-4)
      [[ "$score" = A* ]] && color="green";  [[ "$score" = B* ]] && color="green"
      [[ "$score" = C* ]] && color="yellow"; [[ "$score" = D* ]] && color="orange_2"
      [[ "$score" = E* ]] && color="orange"; [[ "$score" = F* ]] && color="orangered"
      python -m anybadge --label=Maintainability --value="$score" --color="$color" -f build/badges/maintainability -o
  coverage: '/TOTAL.+?(\d+\%)/'
  artifacts:
    when: always
    reports:
      junit: report.xml
      coverage_report:
        coverage_format: cobertura
        path: coverage.xml
    paths:
      - build/coverage
      - build/badges
      - env/
    expire_in: 1w  # most recent artifact is always kept

docs:
  script:
    - pip install -r docs/requirements.txt
    - mkdocs build --clean --site-dir build/docs
  artifacts:
    paths:
      - build/docs

A coverage report is created and linked in the README file.

IDE Integration#

Most Python IDE's integrate with test suites written for pytest and allow you to run them easily during development. Here is an example of VSCode's Testing UI:

VSCode automatically loads your test suite in the "Testing" sidebar and makes it easy to (1) run or debug all your tests or (2) run or debug individual tests. It understands parametrized tests and breaks them out as separate test cases.