Newer
Older
# GitLab CI template for Python
This project implements a generic GitLab CI template for [Python](https://www.python.org/).
It provides several features, usable in different modes (by configuration) following those [recommendations](to-be-continuous.gitlab.io/doc/usage/)
## Usage
In order to include this template in your project, add the following to your `gitlab-ci.yml`:
```yaml
include:
file: '/templates/gitlab-ci-python.yml'
```
## Global configuration
The Python template uses some global configuration used throughout all jobs.
| Name | description | default value |
| -------------------- | ---------------------------------------------------------------------------------------------------------------- | ------------------ |
| `PYTHON_IMAGE` | The Docker image used to run Python. **It is highly recommended to set the specific version your project needs** | `python:3` |
| `PIP_INDEX_URL` | Python repository url | _none_ |
| `PYTHON_PROJECT_DIR` | Python project root directory | `.` |
| `REQUIREMENTS_FILE` | Path to requirements file _(relative to `$PYTHON_PROJECT_DIR`)_ | `requirements.txt` |
| `PIP_OPTS` | pip extra [options](https://pip.pypa.io/en/stable/reference/pip/#general-options) | _none_ |
The cache policy also declares the `.cache/pip` directory as cached (not to download Python dependencies over and over again).
Default configuration follows [this Python project structure](https://docs.python-guide.org/writing/structure/)
### Poetry support
The Python template supports [Poetry](https://python-poetry.org/) as packaging and dependency management tool.
If a `pyproject.toml` and `poetry.lock` file is detected at the root of your project structure, requirements will automatically be generated from Poetry.
Poetry support is disabled if `PYTHON_POETRY_DISABLED` is set to `true`.
:warning: as stated in [Poetry documentation](https://python-poetry.org/docs/basic-usage/#commit-your-poetrylock-file-to-version-control), _You should commit the `poetry.lock` file to your project repo so that all people working on the project are locked to the same versions of dependencies_.
It uses the following variables:
| Name | description | default value |
| ------------------------ | ---------------------------------------------------------- | ----------------- |
| `PYTHON_POETRY_EXTRAS` | Poetry [extra sets of dependencies](https://python-poetry.org/docs/pyproject/#extras) to include, space separated | _none_ |
## Jobs
### Lint jobs
#### `py-pylint` job
This job is **disabled by default** and performs code analysis based on [pylint](http://pylint.pycqa.org/en/latest/) Python lib.
It is activated by setting `$PYLINT_ENABLED` to `true`.
It is bound to the `build` stage, and uses the following variables:
| Name | description | default value |
| ------------------------ | ---------------------------------- | ----------------- |
| `PYLINT_ARGS` | Additional [pylint CLI options](http://pylint.pycqa.org/en/latest/user_guide/run.html#command-line-options) | _none_ |
| `PYLINT_FILES` | Files or directories to analyse | _none_ (by default analyses all found python source files) |
This job produces the following artifacts, kept for one day:
* Code quality json report in code climate format.
### Test jobs
The Python template features four alternative test jobs:
* `py-unittest` that performs tests based on [unittest](https://docs.python.org/3/library/unittest.html) Python lib,
* or `py-pytest` that performs tests based on [pytest](https://docs.pytest.org/en/latest/) Python lib,
* or `py-nosetest` that performs tests based on [nose](https://nose.readthedocs.io/en/latest/) Python lib,
* or `py-compile` that performs byte code generation to check syntax if not tests are available.
#### `py-unittest` job
This job is **disabled by default** and performs tests based on [unittest](https://docs.python.org/3/library/unittest.html) Python lib.
It is activated by setting `$UNITTEST_ENABLED` to `true`.
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
In order to produce JUnit test reports, the tests are executed with the [xmlrunner](https://github.com/xmlrunner/unittest-xml-reporting) module.
It is bound to the `build` stage, and uses the following variables:
| Name | description | default value |
| ------------------------ | -------------------------------------------------------------------- | ----------------------- |
| `TEST_REQUIREMENTS_FILE` | Path to test requirements file _(relative to `$PYTHON_PROJECT_DIR`)_ | `test-requirements.txt` |
| `UNITTEST_ARGS` | Additional xmlrunner/unittest CLI options | _none_ |
This job produces the following artifacts, kept for one day:
* JUnit test report (using the [xmlrunner](https://github.com/xmlrunner/unittest-xml-reporting) module)
* code coverage report (cobertura xml format).
:warning: create a `.coveragerc` file at the root of your Python project to control the coverage settings.
Example:
```conf
[run]
# enables branch coverage
branch = True
# list of directories/packages to cover
source =
module_1
module_2
```
#### `py-pytest` job
This job is **disabled by default** and performs tests based on [pytest](https://docs.pytest.org/en/latest/) Python lib.
It is activated by setting `$PYTEST_ENABLED` to `true`.
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
It is bound to the `build` stage, and uses the following variables:
| Name | description | default value |
| ------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------- |
| `TEST_REQUIREMENTS_FILE` | Path to test requirements file _(relative `$PYTHON_PROJECT_DIR`)_ | `test-requirements.txt` |
| `PYTEST_ARGS` | Additional [pytest](https://docs.pytest.org/en/stable/usage.html) or [pytest-cov](https://github.com/pytest-dev/pytest-cov#usage) CLI options | _none_ |
This job produces the following artifacts, kept for one day:
* JUnit test report (with the [`--junit-xml`](http://doc.pytest.org/en/latest/usage.html#creating-junitxml-format-files) argument)
* code coverage report (cobertura xml format).
:warning: create a `.coveragerc` file at the root of your Python project to control the coverage settings.
Example:
```conf
[run]
# enables branch coverage
branch = True
# list of directories/packages to cover
source =
module_1
module_2
```
#### `py-nosetest` job
This job is **disabled by default** and performs tests based on [nose](https://nose.readthedocs.io/en/latest/) Python lib.
It is activated by setting `$NOSETESTS_ENABLED` to `true`.
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
It is bound to the `build` stage, and uses the following variables:
| Name | description | default value |
| ------------------------ | --------------------------------------------------------------------------------------- | ----------------------- |
| `TEST_REQUIREMENTS_FILE` | Path to test requirements file _(relative to `$PYTHON_PROJECT_DIR`)_ | `test-requirements.txt` |
| `NOSETESTS_ARGS` | Additional [nose CLI options](https://nose.readthedocs.io/en/latest/usage.html#options) | _none_ |
By default coverage will be run on all the directory. You can restrict it to your packages by setting NOSE_COVER_PACKAGE variable.
More [info](https://nose.readthedocs.io/en/latest/plugins/cover.html)
This job produces the following artifacts, kept for one day:
* JUnit test report (with the [`--with-xunit`](https://nose.readthedocs.io/en/latest/plugins/xunit.html) argument)
* code coverage report (cobertura xml format + html report).
:warning: create a `.coveragerc` file at the root of your Python project or use [nose CLI options](https://nose.readthedocs.io/en/latest/plugins/cover.html#options) to control the coverage settings.
#### `py-compile` job
This job is a fallback if no unit test has been setup (`$UNITTEST_ENABLED` and `$PYTEST_ENABLED` and `$NOSETEST_ENABLED`
are not set), and performs a [`compileall`](https://docs.python.org/3/library/compileall.html).
It is bound to the `build` stage, and uses the following variables:
| Name | description | default value |
| --------------------- | ----------------------------------------------------------------------------- | ------------- |
| `PYTHON_COMPILE_ARGS` | [`compileall` CLI options](https://docs.python.org/3/library/compileall.html) | `*` |
### SonarQube analysis
If you're using the SonarQube template to analyse your Python code, here is a sample `sonar-project.properties` file:
```properties
# see: https://docs.sonarqube.org/latest/analysis/languages/python/
# set your source directory(ies) here (relative to the sonar-project.properties file)
sonar.sources=.
# exclude unwanted directories and files from being analysed
sonar.exclusions=**/test_*.py
# set your tests directory(ies) here (relative to the sonar-project.properties file)
sonar.tests=.
sonar.test.inclusions=**/test_*.py
# tests report: generic format
sonar.python.xunit.reportPath=reports/unittest/TEST-*.xml
# coverage report: XUnit format
sonar.python.coverage.reportPaths=reports/coverage.xml
```
More info:
* [Python language support](https://docs.sonarqube.org/latest/analysis/languages/python/)
* [test coverage & execution parameters](https://docs.sonarqube.org/latest/analysis/coverage/)
* [third-party issues](https://docs.sonarqube.org/latest/analysis/external-issues/)
### `py-bandit` job (SAST)
This job is **disabled by default** and performs a [Bandit](https://pypi.org/project/bandit/) analysis.
It is bound to the `test` stage, and uses the following variables:
| Name | description | default value |
| ---------------- | ---------------------------------------------------------------------- | ----------------- |
| `BANDIT_ENABLED` | Set to `true` to enable Bandit analysis | _none_ (disabled) |
| `BANDIT_ARGS` | Additional [Bandit CLI options](https://github.com/PyCQA/bandit#usage) | `--recursive .` |
This job outputs a **textual report** in the console, and in case of failure also exports a JSON report in the `reports/`
directory _(relative to project root dir)_.
### `py-safety` job (dependency check)
This job is **disabled by default** and performs a dependency check analysis using [Safety](https://pypi.org/project/safety/).
It is bound to the `test` stage, and uses the following variables:
| Name | description | default value |
| ---------------- | ----------------------------------------------------------------------- | ----------------- |
| `SAFETY_ENABLED` | Set to `true` to enable Safety job | _none_ (disabled) |
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
| `SAFETY_ARGS` | Additional [Safety CLI options](https://github.com/pyupio/safety#usage) | `--full-report` |
This job outputs a **textual report** in the console, and in case of failure also exports a JSON report in the `reports/`
directory _(relative to project root dir)_.
### Publish jobs
### `py-release` job
This job is **disabled by default** and performs an automatic tagging of your Python code.
* [Bumpversion](https://github.com/peritus/bumpversion) Python library is used for version management.
* Looks for an existing `.bumpversion.cfg` at the project root. If found, it will be the configuration used by bumpversion. If not, the `$RELEASE_VERSION_PART` variable and `setup.py` will be used instead.
* Creating a Git tag involves an authenticated and authorized Git user.
**Don't use your personal password !!!
Use an [access token](https://docs.gitlab.com/ee/user/profile/personal_access_tokens.html) with write_repository rights.
If you have a generic account, add it to the project and generate access token from this account.**
It is bound to the `publish` stage, applies only on master branch and uses the following variables:
| Name | description | default value |
| ---------------------- | ----------------------------------------------------------------------- | ----------------- |
| `RELEASE_VERSION_PART` | The part of the version to increase (one of: `major`, `minor`, `patch`) | `minor` |
| `RELEASE_USERNAME` | Username credential for git push | _none_ (disabled) |
| `RELEASE_ACCESS_TOKEN` | Password credential for git push | _none_ |
### `py-publish` job
This job is **disabled by default** and performs a packaging and publication of your Python code.
It is bound to the `publish` stage, applies only on git tags and uses the following variables:
| Name | description | default value |
| ---------------------- | -------------------------------------------------------- | ----------------- |
| `TWINE_REPOSITORY_URL` | Where to publish your Python project | _none_ (disabled) |
| `TWINE_USERNAME` | Username credential to publish to \$TWINE_REPOSITORY_URL | _none_ (disabled) |
| `TWINE_PASSWORD` | Password credential to publish to \$TWINE_REPOSITORY_URL | _none_ |
More info:
* [Python Packaging User Guide](https://packaging.python.org/)
If you want to automatically create tag and publish your Python package, please have a look [here](#release-python)
### `py-docs` job
This job is **disabled by default** and performs documentation generation of your Python code using [Sphinx](http://www.sphinx-doc.org/en/master/). Documentation will be available through a GitLab artifact.
It is bound to the `publish` stage, applies only on tags and uses the following variables:
| Name | description | default value |
| ------------------------ | -------------------------------------------------------------------------------------- | --------------------------------- |
| `DOCS_ENABLED` | Set to `true` to enable pages job | _none_ (disabled) |
| `DOCS_REQUIREMENTS_FILE` | Python dependencies for documentation generation _(relative to `$PYTHON_PROJECT_DIR`)_ | `docs-requirements.txt` |
| `DOCS_DIRECTORY` | Directory containing docs source | `docs` |
| `DOCS_BUILD_DIR` | Output build directory for documentation | `public` |
| `DOCS_MAKE_ARGS` | Args of make command | `html BUILDDIR=${DOCS_BUILD_DIR}` |