Skip to content
Snippets Groups Projects
README.md 15.7 KiB
Newer Older
Pierre Smeyers's avatar
Pierre Smeyers committed
# GitLab CI template for Python

This project implements a generic GitLab CI template for [Python](https://www.python.org/).

Pierre Smeyers's avatar
Pierre Smeyers committed
It provides several features, usable in different modes (by configuration) following those [recommendations](to-be-continuous.gitlab.io/doc/usage/)
Pierre Smeyers's avatar
Pierre Smeyers committed

## Usage

In order to include this template in your project, add the following to your `gitlab-ci.yml`:

```yaml
include:
Pierre Smeyers's avatar
Pierre Smeyers committed
  - project: 'to-be-continuous/python'
    ref: '1.3.0'
Pierre Smeyers's avatar
Pierre Smeyers committed
    file: '/templates/gitlab-ci-python.yml'
```

## Global configuration

The Python template uses some global configuration used throughout all jobs.

| Name                 | description                                                                                                      | default value      |
| -------------------- | ---------------------------------------------------------------------------------------------------------------- | ------------------ |
| `PYTHON_IMAGE`       | The Docker image used to run Python. **It is highly recommended to set the specific version your project needs** | `python:3`    |
Pierre Smeyers's avatar
Pierre Smeyers committed
| `PIP_INDEX_URL`      | Python repository url                                                                                            | _none_             |
| `PYTHON_PROJECT_DIR` | Python project root directory                                                                                    | `.`                |
| `REQUIREMENTS_FILE`  | Path to requirements file _(relative to `$PYTHON_PROJECT_DIR`)_                                                  | `requirements.txt` |
| `PIP_OPTS`           | pip extra [options](https://pip.pypa.io/en/stable/reference/pip/#general-options)                                | _none_             |

The cache policy also declares the `.cache/pip` directory as cached (not to download Python dependencies over and over again).

Default configuration follows [this Python project structure](https://docs.python-guide.org/writing/structure/)

### Poetry support

The Python template supports [Poetry](https://python-poetry.org/) as packaging and dependency management tool.

If a `pyproject.toml` and `poetry.lock` file is detected at the root of your project structure, requirements will automatically be generated from Poetry.
Poetry support is disabled  if `PYTHON_POETRY_DISABLED` is set to `true`.
Pierre Smeyers's avatar
Pierre Smeyers committed

:warning: as stated in [Poetry documentation](https://python-poetry.org/docs/basic-usage/#commit-your-poetrylock-file-to-version-control), _You should commit the `poetry.lock` file to your project repo so that all people working on the project are locked to the same versions of dependencies_.

It uses the following variables:

| Name                     | description                                                | default value     |
| ------------------------ | ---------------------------------------------------------- | ----------------- |
| `PYTHON_POETRY_EXTRAS`   | Poetry [extra sets of dependencies](https://python-poetry.org/docs/pyproject/#extras) to include, space separated     |  _none_           |

Pierre Smeyers's avatar
Pierre Smeyers committed
## Jobs

### Lint jobs

#### `py-pylint` job

This job is **disabled by default** and performs code analysis based on [pylint](http://pylint.pycqa.org/en/latest/) Python lib.
It is activated by setting `$PYLINT_ENABLED` to `true`.
Pierre Smeyers's avatar
Pierre Smeyers committed

It is bound to the `build` stage, and uses the following variables:

| Name                     | description                        | default value     |
| ------------------------ | ---------------------------------- | ----------------- |
| `PYLINT_ARGS`            | Additional [pylint CLI options](http://pylint.pycqa.org/en/latest/user_guide/run.html#command-line-options)      |  _none_           |
| `PYLINT_FILES`           | Files or directories to analyse   | _none_ (by default analyses all found python source files) |

This job produces the following artifacts, kept for one day:

* Code quality json report in code climate format.

### Test jobs

The Python template features four alternative test jobs:

* `py-unittest` that performs tests based on [unittest](https://docs.python.org/3/library/unittest.html) Python lib,
* or `py-pytest` that performs tests based on [pytest](https://docs.pytest.org/en/latest/) Python lib,
* or `py-nosetest` that performs tests based on [nose](https://nose.readthedocs.io/en/latest/) Python lib,
* or `py-compile` that performs byte code generation to check syntax if not tests are available.

#### `py-unittest` job

This job is **disabled by default** and performs tests based on [unittest](https://docs.python.org/3/library/unittest.html) Python lib.
It is activated by setting `$UNITTEST_ENABLED` to `true`.
Pierre Smeyers's avatar
Pierre Smeyers committed

In order to produce JUnit test reports, the tests are executed with the [xmlrunner](https://github.com/xmlrunner/unittest-xml-reporting) module.

It is bound to the `build` stage, and uses the following variables:

| Name                     | description                                                          | default value           |
| ------------------------ | -------------------------------------------------------------------- | ----------------------- |
| `TEST_REQUIREMENTS_FILE` | Path to test requirements file _(relative to `$PYTHON_PROJECT_DIR`)_ | `test-requirements.txt` |
| `UNITTEST_ARGS`          | Additional xmlrunner/unittest CLI options                            | _none_                  |

This job produces the following artifacts, kept for one day:

* JUnit test report (using the [xmlrunner](https://github.com/xmlrunner/unittest-xml-reporting) module)
* code coverage report (cobertura xml format).

:warning: create a `.coveragerc` file at the root of your Python project to control the coverage settings.

Example:

```conf
[run]
# enables branch coverage
branch = True
# list of directories/packages to cover
source = 
    module_1
    module_2
```

#### `py-pytest` job

This job is **disabled by default** and performs tests based on [pytest](https://docs.pytest.org/en/latest/) Python lib.
It is activated by setting `$PYTEST_ENABLED` to `true`.
Pierre Smeyers's avatar
Pierre Smeyers committed

It is bound to the `build` stage, and uses the following variables:

| Name                     | description                                                                                                                                   | default value           |
| ------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------- |
| `TEST_REQUIREMENTS_FILE` | Path to test requirements file _(relative `$PYTHON_PROJECT_DIR`)_                                                                           | `test-requirements.txt` |
| `PYTEST_ARGS`            | Additional [pytest](https://docs.pytest.org/en/stable/usage.html) or [pytest-cov](https://github.com/pytest-dev/pytest-cov#usage) CLI options | _none_                  |

This job produces the following artifacts, kept for one day:

* JUnit test report (with the [`--junit-xml`](http://doc.pytest.org/en/latest/usage.html#creating-junitxml-format-files) argument)
* code coverage report (cobertura xml format).

:warning: create a `.coveragerc` file at the root of your Python project to control the coverage settings.

Example:

```conf
[run]
# enables branch coverage
branch = True
# list of directories/packages to cover
source = 
    module_1
    module_2
```

#### `py-nosetest` job

This job is **disabled by default** and performs tests based on [nose](https://nose.readthedocs.io/en/latest/) Python lib.
It is activated by setting `$NOSETESTS_ENABLED` to `true`.
Pierre Smeyers's avatar
Pierre Smeyers committed

It is bound to the `build` stage, and uses the following variables:

| Name                     | description                                                                             | default value           |
| ------------------------ | --------------------------------------------------------------------------------------- | ----------------------- |
| `TEST_REQUIREMENTS_FILE` | Path to test requirements file _(relative to `$PYTHON_PROJECT_DIR`)_                    | `test-requirements.txt` |
| `NOSETESTS_ARGS`         | Additional [nose CLI options](https://nose.readthedocs.io/en/latest/usage.html#options) | _none_                  |

By default coverage will be run on all the directory. You can restrict it to your packages by setting NOSE_COVER_PACKAGE variable.
More [info](https://nose.readthedocs.io/en/latest/plugins/cover.html)

This job produces the following artifacts, kept for one day:

* JUnit test report (with the [`--with-xunit`](https://nose.readthedocs.io/en/latest/plugins/xunit.html) argument)
* code coverage report (cobertura xml format + html report).

:warning: create a `.coveragerc` file at the root of your Python project or use [nose CLI options](https://nose.readthedocs.io/en/latest/plugins/cover.html#options) to control the coverage settings.

#### `py-compile` job

This job is a fallback if no unit test has been setup (`$UNITTEST_ENABLED` and `$PYTEST_ENABLED` and `$NOSETEST_ENABLED`
are not set), and performs a [`compileall`](https://docs.python.org/3/library/compileall.html).

It is bound to the `build` stage, and uses the following variables:

| Name                  | description                                                                   | default value |
| --------------------- | ----------------------------------------------------------------------------- | ------------- |
| `PYTHON_COMPILE_ARGS` | [`compileall` CLI options](https://docs.python.org/3/library/compileall.html) | `*`           |

### SonarQube analysis

If you're using the SonarQube template to analyse your Python code, here is a sample `sonar-project.properties` file:

```properties
# see: https://docs.sonarqube.org/latest/analysis/languages/python/
# set your source directory(ies) here (relative to the sonar-project.properties file)
sonar.sources=.
# exclude unwanted directories and files from being analysed
sonar.exclusions=**/test_*.py

# set your tests directory(ies) here (relative to the sonar-project.properties file)
sonar.tests=.
sonar.test.inclusions=**/test_*.py

# tests report: generic format
sonar.python.xunit.reportPath=reports/unittest/TEST-*.xml
# coverage report: XUnit format
sonar.python.coverage.reportPaths=reports/coverage.xml
```

More info:

* [Python language support](https://docs.sonarqube.org/latest/analysis/languages/python/)
* [test coverage & execution parameters](https://docs.sonarqube.org/latest/analysis/coverage/)
* [third-party issues](https://docs.sonarqube.org/latest/analysis/external-issues/)

### `py-bandit` job (SAST)

This job is **disabled by default** and performs a [Bandit](https://pypi.org/project/bandit/) analysis.

It is bound to the `test` stage, and uses the following variables:

| Name             | description                                                            | default value     |
| ---------------- | ---------------------------------------------------------------------- | ----------------- |
| `BANDIT_ENABLED` | Set to `true` to enable Bandit analysis                                     | _none_ (disabled) |
Pierre Smeyers's avatar
Pierre Smeyers committed
| `BANDIT_ARGS`    | Additional [Bandit CLI options](https://github.com/PyCQA/bandit#usage) | `--recursive .`   |

This job outputs a **textual report** in the console, and in case of failure also exports a JSON report in the `reports/`
directory _(relative to project root dir)_.

### `py-safety` job (dependency check)

This job is **disabled by default** and performs a dependency check analysis using [Safety](https://pypi.org/project/safety/).

It is bound to the `test` stage, and uses the following variables:

| Name             | description                                                             | default value     |
| ---------------- | ----------------------------------------------------------------------- | ----------------- |
| `SAFETY_ENABLED` | Set to `true` to enable Safety job                                           | _none_ (disabled) |
Pierre Smeyers's avatar
Pierre Smeyers committed
| `SAFETY_ARGS`    | Additional [Safety CLI options](https://github.com/pyupio/safety#usage) | `--full-report`   |

This job outputs a **textual report** in the console, and in case of failure also exports a JSON report in the `reports/`
directory _(relative to project root dir)_.

### Publish jobs

### `py-release` job

This job is **disabled by default** and performs an automatic tagging of your Python code.

* [Bumpversion](https://github.com/peritus/bumpversion) Python library is used for version management.
* Looks for an existing `.bumpversion.cfg` at the project root. If found, it will be the configuration used by bumpversion. If not, the `$RELEASE_VERSION_PART` variable and `setup.py` will be used instead.
* Creating a Git tag involves an authenticated and authorized Git user.

**Don't use your personal password !!!
Use an [access token](https://docs.gitlab.com/ee/user/profile/personal_access_tokens.html) with write_repository rights.
If you have a generic account, add it to the project and generate access token from this account.**

It is bound to the `publish` stage, applies only on master branch and uses the following variables:

| Name                   | description                                                             | default value     |
| ---------------------- | ----------------------------------------------------------------------- | ----------------- |
| `RELEASE_VERSION_PART` | The part of the version to increase (one of: `major`, `minor`, `patch`) | `minor`           |
| `RELEASE_USERNAME`     | Username credential for git push                                        | _none_ (disabled) |
| `RELEASE_ACCESS_TOKEN` | Password credential for git push                                        | _none_            |

### `py-publish` job

This job is **disabled by default** and performs a packaging and publication of your Python code.

It is bound to the `publish` stage, applies only on git tags and uses the following variables:

| Name                   | description                                              | default value     |
| ---------------------- | -------------------------------------------------------- | ----------------- |
| `TWINE_REPOSITORY_URL` | Where to publish your Python project                     | _none_ (disabled) |
| `TWINE_USERNAME`       | Username credential to publish to \$TWINE_REPOSITORY_URL | _none_ (disabled) |
| `TWINE_PASSWORD`       | Password credential to publish to \$TWINE_REPOSITORY_URL | _none_            |

More info:

* [Python Packaging User Guide](https://packaging.python.org/)

If you want to automatically create tag and publish your Python package, please have a look [here](#release-python)

### `py-docs` job

This job is **disabled by default** and performs documentation generation of your Python code using [Sphinx](http://www.sphinx-doc.org/en/master/). Documentation will be available through a GitLab artifact.

It is bound to the `publish` stage, applies only on tags and uses the following variables:

| Name                     | description                                                                            | default value                     |
| ------------------------ | -------------------------------------------------------------------------------------- | --------------------------------- |
| `DOCS_ENABLED`           | Set to `true` to enable pages job                                                           | _none_ (disabled)                 |
Pierre Smeyers's avatar
Pierre Smeyers committed
| `DOCS_REQUIREMENTS_FILE` | Python dependencies for documentation generation _(relative to `$PYTHON_PROJECT_DIR`)_ | `docs-requirements.txt`           |
| `DOCS_DIRECTORY`         | Directory containing docs source                                                       | `docs`                            |
| `DOCS_BUILD_DIR`         | Output build directory for documentation                                               | `public`                          |
| `DOCS_MAKE_ARGS`         | Args of make command                                                                   | `html BUILDDIR=${DOCS_BUILD_DIR}` |