# GitLab CI template for Python
This project implements a generic GitLab CI template for [Python](https://www.python.org/).
It provides several features, usable in different modes (by configuration) following those [recommendations](to-be-continuous.gitlab.io/doc/usage/)
## Usage
In order to include this template in your project, add the following to your `gitlab-ci.yml`:
```yaml
include:
- project: 'to-be-continuous/python'
ref: '2.1.1'
file: '/templates/gitlab-ci-python.yml'
```
## Global configuration
The Python template uses some global configuration used throughout all jobs.
| Name | description | default value |
| -------------------- | ------------------------------------------------------------------------------------- | ------------------ |
| `PYTHON_IMAGE` | The Docker image used to run Python <br/>:warning: **set the version required by your project** | `python:3` |
| `PIP_INDEX_URL` | Python repository url | _none_ |
| `PYTHON_PROJECT_DIR` | Python project root directory | `.` |
| `REQUIREMENTS_FILE` | Path to requirements file _(relative to `$PYTHON_PROJECT_DIR`)_ | `requirements.txt` |
| `PIP_OPTS` | pip extra [options](https://pip.pypa.io/en/stable/reference/pip/#general-options) | _none_ |
The cache policy also declares the `.cache/pip` directory as cached (not to download Python dependencies over and over again).
Default configuration follows [this Python project structure](https://docs.python-guide.org/writing/structure/)
### Poetry support
The Python template supports [Poetry](https://python-poetry.org/) as packaging and dependency management tool.
If a `pyproject.toml` and `poetry.lock` file is detected at the root of your project structure, requirements will automatically be generated from Poetry.
Poetry support is disabled if `PYTHON_POETRY_DISABLED` is set to `true`.
:warning: as stated in [Poetry documentation](https://python-poetry.org/docs/basic-usage/#commit-your-poetrylock-file-to-version-control), _You should commit the `poetry.lock` file to your project repo so that all people working on the project are locked to the same versions of dependencies_.
It uses the following variables:
| Name | description | default value |
| ------------------------ | ---------------------------------------------------------- | ----------------- |
| `PYTHON_POETRY_EXTRAS` | Poetry [extra sets of dependencies](https://python-poetry.org/docs/pyproject/#extras) to include, space separated | _none_ |
## Jobs
### Lint jobs
#### `py-pylint` job
This job is **disabled by default** and performs code analysis based on [pylint](http://pylint.pycqa.org/en/latest/) Python lib.
It is activated by setting `$PYLINT_ENABLED` to `true`.
It is bound to the `build` stage, and uses the following variables:
| Name | description | default value |
| ------------------------ | ---------------------------------- | ----------------- |
| `PYLINT_ARGS` | Additional [pylint CLI options](http://pylint.pycqa.org/en/latest/user_guide/run.html#command-line-options) | _none_ |
| `PYLINT_FILES` | Files or directories to analyse | _none_ (by default analyses all found python source files) |
This job produces the following artifacts, kept for one day:
* Code quality json report in code climate format.
### Test jobs
The Python template features four alternative test jobs:
* `py-unittest` that performs tests based on [unittest](https://docs.python.org/3/library/unittest.html) Python lib,
* or `py-pytest` that performs tests based on [pytest](https://docs.pytest.org/en/latest/) Python lib,
* or `py-nosetest` that performs tests based on [nose](https://nose.readthedocs.io/en/latest/) Python lib,
* or `py-compile` that performs byte code generation to check syntax if not tests are available.
#### `py-unittest` job
This job is **disabled by default** and performs tests based on [unittest](https://docs.python.org/3/library/unittest.html) Python lib.
It is activated by setting `$UNITTEST_ENABLED` to `true`.
In order to produce JUnit test reports, the tests are executed with the [xmlrunner](https://github.com/xmlrunner/unittest-xml-reporting) module.
It is bound to the `build` stage, and uses the following variables:
| Name | description | default value |
| ------------------------ | -------------------------------------------------------------------- | ----------------------- |
| `TEST_REQUIREMENTS_FILE` | Path to test requirements file _(relative to `$PYTHON_PROJECT_DIR`)_ | `test-requirements.txt` |
| `UNITTEST_ARGS` | Additional xmlrunner/unittest CLI options | _none_ |
This job produces the following artifacts, kept for one day:
* JUnit test report (using the [xmlrunner](https://github.com/xmlrunner/unittest-xml-reporting) module)
* code coverage report (cobertura xml format).
:warning: create a `.coveragerc` file at the root of your Python project to control the coverage settings.
Example:
```conf
[run]
# enables branch coverage
branch = True
# list of directories/packages to cover
source =
module_1
module_2
```
#### `py-pytest` job
This job is **disabled by default** and performs tests based on [pytest](https://docs.pytest.org/en/latest/) Python lib.
It is activated by setting `$PYTEST_ENABLED` to `true`.
It is bound to the `build` stage, and uses the following variables:
| Name | description | default value |
| ------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------- |
| `TEST_REQUIREMENTS_FILE` | Path to test requirements file _(relative `$PYTHON_PROJECT_DIR`)_ | `test-requirements.txt` |
| `PYTEST_ARGS` | Additional [pytest](https://docs.pytest.org/en/stable/usage.html) or [pytest-cov](https://github.com/pytest-dev/pytest-cov#usage) CLI options | _none_ |
This job produces the following artifacts, kept for one day:
* JUnit test report (with the [`--junit-xml`](http://doc.pytest.org/en/latest/usage.html#creating-junitxml-format-files) argument)
* code coverage report (cobertura xml format).
:warning: create a `.coveragerc` file at the root of your Python project to control the coverage settings.
Example:
```conf
[run]
# enables branch coverage
branch = True
# list of directories/packages to cover
source =
module_1
module_2
```
#### `py-nosetest` job
This job is **disabled by default** and performs tests based on [nose](https://nose.readthedocs.io/en/latest/) Python lib.
It is activated by setting `$NOSETESTS_ENABLED` to `true`.
It is bound to the `build` stage, and uses the following variables:
| Name | description | default value |
| ------------------------ | --------------------------------------------------------------------------------------- | ----------------------- |
| `TEST_REQUIREMENTS_FILE` | Path to test requirements file _(relative to `$PYTHON_PROJECT_DIR`)_ | `test-requirements.txt` |
| `NOSETESTS_ARGS` | Additional [nose CLI options](https://nose.readthedocs.io/en/latest/usage.html#options) | _none_ |
By default coverage will be run on all the directory. You can restrict it to your packages by setting NOSE_COVER_PACKAGE variable.
More [info](https://nose.readthedocs.io/en/latest/plugins/cover.html)
This job produces the following artifacts, kept for one day:
* JUnit test report (with the [`--with-xunit`](https://nose.readthedocs.io/en/latest/plugins/xunit.html) argument)
* code coverage report (cobertura xml format + html report).
:warning: create a `.coveragerc` file at the root of your Python project or use [nose CLI options](https://nose.readthedocs.io/en/latest/plugins/cover.html#options) to control the coverage settings.
#### `py-compile` job
This job is a fallback if no unit test has been setup (`$UNITTEST_ENABLED` and `$PYTEST_ENABLED` and `$NOSETEST_ENABLED`
are not set), and performs a [`compileall`](https://docs.python.org/3/library/compileall.html).
It is bound to the `build` stage, and uses the following variables:
| Name | description | default value |
| --------------------- | ----------------------------------------------------------------------------- | ------------- |
| `PYTHON_COMPILE_ARGS` | [`compileall` CLI options](https://docs.python.org/3/library/compileall.html) | `*` |
### SonarQube analysis
If you're using the SonarQube template to analyse your Python code, here is a sample `sonar-project.properties` file:
```properties
# see: https://docs.sonarqube.org/latest/analysis/languages/python/
# set your source directory(ies) here (relative to the sonar-project.properties file)
sonar.sources=.
# exclude unwanted directories and files from being analysed
sonar.exclusions=**/test_*.py
# set your tests directory(ies) here (relative to the sonar-project.properties file)
sonar.tests=.
sonar.test.inclusions=**/test_*.py
# tests report: generic format
sonar.python.xunit.reportPath=reports/unittest/TEST-*.xml
# coverage report: XUnit format
sonar.python.coverage.reportPaths=reports/coverage.xml
```
More info:
* [Python language support](https://docs.sonarqube.org/latest/analysis/languages/python/)
* [test coverage & execution parameters](https://docs.sonarqube.org/latest/analysis/coverage/)
* [third-party issues](https://docs.sonarqube.org/latest/analysis/external-issues/)
### `py-bandit` job (SAST)
This job is **disabled by default** and performs a [Bandit](https://pypi.org/project/bandit/) analysis.
It is bound to the `test` stage, and uses the following variables:
| Name | description | default value |
| ---------------- | ---------------------------------------------------------------------- | ----------------- |
| `BANDIT_ENABLED` | Set to `true` to enable Bandit analysis | _none_ (disabled) |
| `BANDIT_ARGS` | Additional [Bandit CLI options](https://github.com/PyCQA/bandit#usage) | `--recursive .` |
This job outputs a **textual report** in the console, and in case of failure also exports a JSON report in the `reports/`
directory _(relative to project root dir)_.
### `py-safety` job (dependency check)
This job is **disabled by default** and performs a dependency check analysis using [Safety](https://pypi.org/project/safety/).
It is bound to the `test` stage, and uses the following variables:
| Name | description | default value |
| ---------------- | ----------------------------------------------------------------------- | ----------------- |
| `SAFETY_ENABLED` | Set to `true` to enable Safety job | _none_ (disabled) |
| `SAFETY_ARGS` | Additional [Safety CLI options](https://github.com/pyupio/safety#usage) | `--full-report` |
This job outputs a **textual report** in the console, and in case of failure also exports a JSON report in the `reports/`
directory _(relative to project root dir)_.
### Publish jobs
#### `py-release` job
This job is **disabled by default** and performs an automatic tagging of your Python code.
* [Bumpversion](https://github.com/peritus/bumpversion) Python library is used for version management.
* Looks for an existing `.bumpversion.cfg` at the project root. If found, it will be the configuration used by bumpversion. If not, the `$RELEASE_VERSION_PART` variable and `setup.py` will be used instead.
* Creating a Git tag involves an authenticated and authorized Git user.
**Don't use your personal password !!!
Use an [access token](https://docs.gitlab.com/ee/user/profile/personal_access_tokens.html) with write_repository rights.
If you have a generic account, add it to the project and generate access token from this account.**
It is bound to the `publish` stage, applies only on master branch and uses the following variables:
| Name | description | default value |
| ---------------------- | ----------------------------------------------------------------------- | ----------------- |
| `RELEASE_VERSION_PART` | The part of the version to increase (one of: `major`, `minor`, `patch`) | `minor` |
| `RELEASE_USERNAME` | Username credential for git push | _none_ (disabled) |
| `RELEASE_ACCESS_TOKEN` | Password credential for git push | _none_ |
#### `py-publish` job
This job is **disabled by default** and performs a packaging and publication of your Python code.
It is bound to the `publish` stage, applies only on git tags and uses the following variables:
| Name | description | default value |
| ---------------------- | -------------------------------------------------------- | ----------------- |
| `TWINE_REPOSITORY_URL` | Where to publish your Python project | _none_ (disabled) |
| `TWINE_USERNAME` | Username credential to publish to \$TWINE_REPOSITORY_URL | _none_ (disabled) |
| `TWINE_PASSWORD` | Password credential to publish to \$TWINE_REPOSITORY_URL | _none_ |
More info:
* [Python Packaging User Guide](https://packaging.python.org/)
If you want to automatically create tag and publish your Python package, please have a look [here](#release-python)
#### `py-docs` job
This job is **disabled by default** and performs documentation generation of your Python code using [Sphinx](http://www.sphinx-doc.org/en/master/). Documentation will be available through a GitLab artifact.
It is bound to the `publish` stage, applies only on tags and uses the following variables:
| Name | description | default value |
| ------------------------ | -------------------------------------------------------------------------------------- | --------------------------------- |
| `DOCS_ENABLED` | Set to `true` to enable pages job | _none_ (disabled) |
| `DOCS_REQUIREMENTS_FILE` | Python dependencies for documentation generation _(relative to `$PYTHON_PROJECT_DIR`)_ | `docs-requirements.txt` |
| `DOCS_DIRECTORY` | Directory containing docs source | `docs` |
| `DOCS_BUILD_DIR` | Output build directory for documentation | `public` |
| `DOCS_MAKE_ARGS` | Args of make command | `html BUILDDIR=${DOCS_BUILD_DIR}` |
## GitLab compatibility
:information_source: This template is actually tested and validated on GitLab Community Edition instance version 13.12.11