From 8129ecd0b322e97d4a6db981196cb140b05af49d Mon Sep 17 00:00:00 2001
From: Pierre Smeyers <pierre.smeyers@gmail.com>
Date: Sat, 17 Dec 2022 10:20:18 +0100
Subject: [PATCH] doc: reorganise chapters
---
README.md | 233 ++++++++++++++++++++++++++++++++----------------------
1 file changed, 138 insertions(+), 95 deletions(-)
diff --git a/README.md b/README.md
index 2410f99..ecc6712 100644
--- a/README.md
+++ b/README.md
@@ -2,14 +2,29 @@
This project implements a generic GitLab CI template for [Helm](https://helm.sh/).
-## Overview
+## Usage
+
+In order to include this template in your project, add the following to your `gitlab-ci.yml`:
+
+```yaml
+include:
+ - project: 'to-be-continuous/helm'
+ ref: '3.2.0'
+ file: '/templates/gitlab-ci-helm.yml'
+```
+
+## Understand
+
+This chapter introduces key notions and principle to understand how this template works.
+
+### Managed deployment environments
This template implements continuous delivery/continuous deployment based on [Helm](https://helm.sh/) for projects hosted
on [Kubernetes](https://kubernetes.io) platforms.
It provides several features, usable in different modes (by configuration).
-### Review environments
+#### Review environments
The template supports **review** environments: those are dynamic and ephemeral environments to deploy your
_ongoing developments_ (a.k.a. _feature_ or _topic_ branches).
@@ -21,7 +36,7 @@ It is a strict equivalent of GitLab's [Review Apps](https://docs.gitlab.com/ee/c
It also comes with a _cleanup_ job (accessible either from the _environments_ page, or from the pipeline view).
-### Integration environment
+#### Integration environment
If you're using a Git Workflow with an integration branch (such as [Gitflow](https://www.atlassian.com/git/tutorials/comparing-workflows/gitflow-workflow)),
the template supports an **integration** environment.
@@ -29,7 +44,7 @@ the template supports an **integration** environment.
When enabled, it deploys the result from upstream build stages to a dedicated environment.
It is only active for your integration branch (`develop` by default).
-### Production environments
+#### Production environments
Lastly, the template supports 2 environments associated to your production branch (`master` by default):
@@ -41,51 +56,106 @@ You're free to enable whichever or both, and you can also choose your deployment
* **continuous deployment**: automatic deployment to production (when the upstream pipeline is successful),
* **continuous delivery**: deployment to production can be triggered manually (when the upstream pipeline is successful).
-## Usage
+<!--
+### Supported authentication methods
+-->
-### Include
+### Deployment context variables
-In order to include this template in your project, add the following to your `gitlab-ci.yml`:
+In order to manage the various deployment environments, this template provides a couple of **dynamic variables**
+that you might use in your hook scripts and Helm charts (as [values](https://helm.sh/docs/chart_best_practices/values/)):
-```yaml
-include:
- - project: 'to-be-continuous/helm'
- ref: '3.2.0'
- file: '/templates/gitlab-ci-helm.yml'
-```
+| environment variable | template directive | description |
+|----------------------|--------------------|-------------|
+| `$environment_name` | `{{ .Release.Name }}` | a generated application name to use for the current deployment environment (ex: `myproject-review-fix-bug-12` or `myproject-staging`). This is used as the **Helm release name** in deploy & delete jobs - _details below_ |
+| `$environment_type` | `{{ .Values.environment_type }}` | the current deployment environment type (`review`, `integration`, `staging` or `production`) |
+| `$hostname` | `{{ .Values.hostname }}` | the environment hostame, if you [specified the environment url statically](#environments-url-management) |
-### Global configuration
+#### Generated environment name
-The Helm template uses some global configuration used throughout all jobs.
+The `${environment_name}` variable is generated to designate each deployment environment with a unique and meaningful application name.
+By construction, it is suitable for inclusion in DNS, URLs, Kubernetes labels...
+It is built from:
-| Name | description | default value |
-| --------------------- | -------------------------------------- | ----------------- |
-| `HELM_CLI_IMAGE` | The Docker image used to run Helm <br/>:warning: **set the version required by your Kubernetes server** | `alpine/helm:latest` |
-| `HELM_CHART_DIR` | The folder where the Helm chart is located | `.` _(root project dir)_ |
-| `HELM_SCRIPTS_DIR` | The folder where hook scripts are located | `.` _(root project dir)_ |
-| `HELM_COMMON_VALUES` | Common values file (used for all environments, overridden by specific per-env values files) | undefined (none) |
-| `HELM_ENV_VALUE_NAME` | The environment type variable set to helm | `environment_type` |
-| `HELM_HOSTNAME_VALUE_NAME` | The hostname variable set to helm | `hostname` |
+* the application _base name_ (defaults to `$CI_PROJECT_NAME` but can be overridden globally and/or per deployment environment - _see configuration variables_)
+* GitLab predefined `$CI_ENVIRONMENT_SLUG` variable ([sluggified](https://en.wikipedia.org/wiki/Clean_URL#Slug) name, truncated to 24 characters)
-### Charts publishing
+The `${environment_name}` variable is then evaluated as:
-The template builds a chart package that may be pushed as two distinct packages, depending on a certain _workflow_:
+* `<app base name>` for the production environment
+* `<app base name>-$CI_ENVIRONMENT_SLUG` for all other deployment environments
+* :bulb: `${environment_name}` can also be overriden per environment with the appropriate configuration variable
-1. **snapshot**: the chart is first packaged and then pushed to some registry as
- the **snapshot** image. It can be seen as the raw result of the build, but still **untested and unreliable**.
-2. **release**: once the snapshot chart has been thoroughly tested (both by `package-test` stage jobs and/or `acceptance`
- stage jobs after being deployed to some server), then the chart is pushed one more time as the **release** chart.
- This second push can be seen as the **promotion** of the snapshot chart being now **tested and reliable**.
+Examples (with an application's base name `myapp`):
-Common variables for `helm-package` and `helm-pusblish`:
+| `$environment_type` | Branch | `$CI_ENVIRONMENT_SLUG` | `$environment_name` |
+|---------------------|---------------|-------------------------|---------------------|
+| `review` | `feat/blabla` | `review-feat-bla-xmuzs6`| `myapp-review-feat-bla-xmuzs6` |
+| `integration` | `develop` | `integration` | `myapp-integration` |
+| `staging` | `main` | `staging` | `myapp-staging` |
+| `production` | `main` | `production` | `myapp` |
-| Name | description | default value |
-| --------------------------------- | -------------------------------------------- | ----------------------- |
-| `HELM_REPO_PUBLISH_METHOD` | HTTP method to use to push the package | `POST` |
-| :lock: `HELM_REPO_USER` | Helm registry username | `$CI_REGISTRY_USER` |
-| :lock: `HELM_REPO_PASSWORD` | Helm registry password | `$CI_REGISTRY_PASSWORD` |
+### Deployment and cleanup scripts
+
+The Helm template requires you to provide a Helm chart (either in the project or located in an external repository) to deploy and delete the application.
+
+The environment deployment is processed as follows:
+
+1. _optionally_ executes the `helm-pre-deploy.sh` script in your project to perform specific environment pre-initialization (for e.g. create required services),
+2. [`helm upgrade`](https://helm.sh/docs/helm/helm_upgrade/) the chart with the configured parameters, using [`$environment_name`](#using-variables) as release name,
+3. _optionally_ executes the `helm-post-deploy.sh` script in your project to perform specific environment post-initialization stuff,
+
+The environment deletion is processed as follows:
+
+1. _optionally_ executes the `helm-pre-delete.sh` script in your project to perform specific environment pre-cleanup stuff,
+2. [`helm uninstall`](https://helm.sh/docs/helm/helm_uninstall/), using [`$environment_name`](#using-variables) as release name,
+3. _optionally_ executes the `helm-post-delete.sh` script in your project to perform specific environment post-cleanup (for e.g. delete bound services).
+
+:warning: each of the above hook scripts needs to be executable, you can add flag execution with: `git update-index --chmod=+x helm-pre-cleanup.sh`
+
+### Environments URL management
+
+The Helm template supports two ways of providing your environments url:
+
+* a **static way**: when the environments url can be determined in advance, probably because you're exposing your routes through a DNS you manage,
+* a [**dynamic way**](https://docs.gitlab.com/ee/ci/environments/#set-dynamic-environment-urls-after-a-job-finishes): when the url cannot be known before the
+ deployment job is executed.
+
+The **static way** can be implemented simply by setting the appropriate configuration variable(s) depending on the environment (see environments configuration chapters):
+
+* `$HELM_ENVIRONMENT_URL` to define a default url pattern for all your envs,
+* `$HELM_REVIEW_ENVIRONMENT_URL`, `$HELM_INTEG_ENVIRONMENT_URL`, `$HELM_STAGING_ENVIRONMENT_URL` and `$HELM_PROD_ENVIRONMENT_URL` to override the default.
+
+> :information_source: Each of those variables support a **late variable expansion mechanism** with the `%{somevar}` syntax,
+> allowing you to use any dynamically evaluated variables such as `${environment_name}`.
+>
+> Example:
+>
+> ```yaml
+> variables:
+> HELM_BASE_APP_NAME: "wonderapp"
+> # global url for all environments
+> HELM_ENVIRONMENT_URL: "https://%{environment_name}.nonprod.acme.domain"
+> # override for prod (late expansion of $HELM_BASE_APP_NAME not needed here)
+> HELM_PROD_ENVIRONMENT_URL: "https://$HELM_BASE_APP_NAME.acme.domain"
+> # override for review (using separate resource paths)
+> HELM_REVIEW_ENVIRONMENT_URL: "https://wonderapp-review.nonprod.acme.domain/%{environment_name}"
+> ```
+
+To implement the **dynamic way**, your deployment script shall simply generate a `environment_url.txt` file in the working directory, containing only
+the dynamically generated url. When detected by the template, it will use it as the newly deployed environment url.
+
+### Deployment output variables
+
+Each deployment job produces _output variables_ that are propagated to downstream jobs (using [dotenv artifacts](https://docs.gitlab.com/ee/ci/pipelines/job_artifacts.html#artifactsreportsdotenv)):
+* `$environment_type`: set to the type of environment (`review`, `integration`, `staging` or `production`),
+* `$environment_name`: the application name (see below),
+* `$environment_url`: set to the environment URL (whether determined statically or dynamically).
+Those variables may be freely used in downstream jobs (for instance to run acceptance tests against the latest deployed environment).
+
+## Configuration reference
### Secrets management
@@ -101,22 +171,18 @@ Here are some advices about your **secrets** (variables marked with a :lock:):
it will then be possible to mask it and the template will automatically decode it prior to using it.
3. Don't forget to escape special characters (ex: `$` -> `$$`).
-:warning: your [Values files](https://helm.sh/docs/chart_template_guide/values_files/) **may** contain variable patterns such as `${MY_SECRET}`.
-If so, those patterns will be evaluated (replaced) with actual environment values. This is a safe way of managing your application secrets.
-
-### Deploy & cleanup jobs
-
-The Helm template declares deployment & cleanup jobs for each supported environment.
-
-It supports 2 deployment cases:
-
-* using an **external** Helm chart (retrieved from a repository),
-* using an **internal** Helm chart (located in the project).
+### Global configuration
-Here are global configuration variables for deploy jobs.
+The Helm template uses some global configuration used throughout all jobs.
| Name | description | default value |
| --------------------- | -------------------------------------- | ----------------- |
+| `HELM_CLI_IMAGE` | The Docker image used to run Helm <br/>:warning: **set the version required by your Kubernetes server** | `alpine/helm:latest` |
+| `HELM_CHART_DIR` | The folder where the Helm chart is located | `.` _(root project dir)_ |
+| `HELM_SCRIPTS_DIR` | The folder where hook scripts are located | `.` _(root project dir)_ |
+| `HELM_COMMON_VALUES` | Common values file (used for all environments, overridden by specific per-env values files) | undefined (none) |
+| `HELM_ENV_VALUE_NAME` | The environment type variable set to helm | `environment_type` |
+| `HELM_HOSTNAME_VALUE_NAME` | The hostname variable set to helm | `hostname` |
| `KUBE_NAMESPACE` | The default Kubernetes namespace to use | _none_ but this variable is automatically set by [GitLab Kubernetes integration](https://docs.gitlab.com/ee/user/project/clusters/index.html) when enabled |
| :lock: `HELM_DEFAULT_KUBE_CONFIG` | The default kubeconfig content to use | `$KUBECONFIG` (thus supports the [GitLab Kubernetes integration](https://docs.gitlab.com/ee/user/project/clusters/index.html) when enabled) |
| `HELM_DEPLOY_ARGS` | The Helm [command with options](https://helm.sh/docs/helm/helm_upgrade/) to deploy the application (_without dynamic arguments such as release name and chart_) | `upgrade --install --atomic --timeout 120s` |
@@ -126,17 +192,7 @@ Here are global configuration variables for deploy jobs.
| `HELM_BASE_APP_NAME` | Base application name | `$CI_PROJECT_NAME` ([see GitLab doc](https://docs.gitlab.com/ee/ci/variables/predefined_variables.html)) |
|Â `HELM_ENVIRONMENT_URL` | Default environments url _(only define for static environment URLs declaration)_<br/>_supports late variable expansion (ex: `https://%{environment_name}.helm.acme.com`)_ | _none_ |
-Each deployment job produces _output variables_ that are propagated to downstream jobs (using [dotenv artifacts](https://docs.gitlab.com/ee/ci/pipelines/job_artifacts.html#artifactsreportsdotenv)):
-
-* `environment_type`: set to the type of environment (`review`, `integration`, `staging` or `production`),
-* `environment_name`: the application name (see below),
-* `environment_url`: set to `$CI_ENVIRONMENT_URL`.
-
-They may be freely used in downstream jobs (for instance to run acceptance tests against the latest deployed environment).
-
-Here are configuration details for each environment.
-
-#### Review environments
+### Review environments configuration
Review environments are dynamic and ephemeral environments to deploy your _ongoing developments_ (a.k.a. _feature_ or _topic_ branches).
@@ -153,7 +209,7 @@ Here are variables supported to configure review environments:
| :lock: `HELM_REVIEW_KUBE_CONFIG` | kubeconfig content used for `review` env _(only define to override default)_ | `$HELM_DEFAULT_KUBE_CONFIG` |
| `HELM_REVIEW_VALUES` | The [Values file](https://helm.sh/docs/chart_template_guide/values_files/) to use with `review` environments | _none_ |
-#### Integration environment
+### Integration environment configuration
The integration environment is the environment associated to your integration branch (`develop` by default).
@@ -170,7 +226,7 @@ Here are variables supported to configure the integration environment:
| :lock: `HELM_INTEG_KUBE_CONFIG` | kubeconfig content used for `integration` env _(only define to override default)_ | `$HELM_DEFAULT_KUBE_CONFIG` |
| `HELM_INTEG_VALUES` | The [Values file](https://helm.sh/docs/chart_template_guide/values_files/) to use with the `integration` environment | _none_ |
-#### Staging environment
+### Staging environment configuration
The staging environment is an iso-prod environment meant for testing and validation purpose associated to your production branch (`master` by default).
@@ -187,7 +243,7 @@ Here are variables supported to configure the staging environment:
| :lock: `HELM_STAGING_KUBE_CONFIG` | kubeconfig content used for `staging` env _(only define to override default)_ | `$HELM_DEFAULT_KUBE_CONFIG` |
| `HELM_STAGING_VALUES` | The [Values file](https://helm.sh/docs/chart_template_guide/values_files/) to use with the staging environment | _none_ |
-#### Production environment
+### Production environment configuration
The production environment is the final deployment environment associated with your production branch (`master` by default).
@@ -205,37 +261,6 @@ Here are variables supported to configure the production environment:
| `AUTODEPLOY_TO_PROD` | Set this variable to auto-deploy to production. If not set deployment to production will be `manual` (default behaviour). | _none_ (disabled) |
| `HELM_PROD_VALUES` | The [Values file](https://helm.sh/docs/chart_template_guide/values_files/) to use with the production environment | _none_ |
-#### Dynamic Values and Variables
-
-You have to be aware that your deployment (and cleanup) scripts have to be able to cope with various environments
-(`review`, `integration`, `staging` and `production`), each with different application names, exposed routes, settings, ...
-
-Part of this complexity can be handled by the lookup policies described above (ex: one resource per env).
-
-In order to be able to implement some **genericity** in your scripts and templates:
-
-1. you should use generic [values](https://helm.sh/docs/chart_best_practices/values/) dynamically set and passed by the template:
-
-* `$HELM_ENV_VALUE_NAME` (set by default to `environment_type`): the environment type (`review`, `integration`, `staging` or `production`)
-* `$HELM_HOSTNAME_VALUE_NAME` (set by default to `hostname`): the environment hostname, extracted from `${CI_ENVIRONMENT_URL}` (got from [`environment:url`](https://docs.gitlab.com/ee/ci/yaml/#environmenturl) - see `OS_REVIEW_ENVIRONMENT_SCHEME`, `OS_REVIEW_ENVIRONMENT_DOMAIN`, `OS_STAGING_ENVIRONMENT_URL` and `OS_PROD_ENVIRONMENT_URL`)
-
-2. you should use available environment variables:
-
- * any [GitLab CI variable](https://docs.gitlab.com/ee/ci/variables/predefined_variables.html)
- (ex: `${CI_ENVIRONMENT_URL}` to retrieve the actual environment exposed route)
- * any [custom variable](https://docs.gitlab.com/ee/ci/variables/#add-a-cicd-variable-to-a-project)
- (ex: `${SECRET_TOKEN}` that you have set in your project CI/CD variables)
-
-> :warning:
->
-> In order to be properly replaced, variables in your YAML value file shall be written with curly braces (ex: `${MYVAR}` and not `$MYVAR`).
->
-> Multiline variables must be surrounded by double quotes and you might have to disable line-length rule of yamllint as they are rewritten on a single line.
->
-> ```yaml
-> tlsKey: "${MYKEY}" # yamllint disable-line rule:line-length
-> ```
-
### `helm-lint` job
This job [examines your chart for possible issues](https://helm.sh/docs/helm/helm_lint/) and uses the following variables:
@@ -267,7 +292,25 @@ This job runs [Kube-Score](https://kube-score.com/) on the resources to be creat
| `HELM_KUBE_SCORE_IMAGE` | The Docker image used to run [Kube-Score](https://kube-score.com/) | `zegl/kube-score:latest-helm3` |
| `HELM_KUBE_SCORE_ARGS` | Arguments used by the helm-score job | _none_ |
-### `helm-package` job
+### Charts publishing
+
+The template builds a chart package that may be pushed as two distinct packages, depending on a certain _workflow_:
+
+1. **snapshot**: the chart is first packaged and then pushed to some registry as
+ the **snapshot** image. It can be seen as the raw result of the build, but still **untested and unreliable**.
+2. **release**: once the snapshot chart has been thoroughly tested (both by `package-test` stage jobs and/or `acceptance`
+ stage jobs after being deployed to some server), then the chart is pushed one more time as the **release** chart.
+ This second push can be seen as the **promotion** of the snapshot chart being now **tested and reliable**.
+
+Common variables for `helm-package` and `helm-pusblish`:
+
+| Name | description | default value |
+| --------------------------------- | -------------------------------------------- | ----------------------- |
+| `HELM_REPO_PUBLISH_METHOD` | HTTP method to use to push the package | `POST` |
+| :lock: `HELM_REPO_USER` | Helm registry username | `$CI_REGISTRY_USER` |
+| :lock: `HELM_REPO_PASSWORD` | Helm registry password | `$CI_REGISTRY_PASSWORD` |
+
+#### `helm-package` job
This job [packages your chart into an archive](https://helm.sh/docs/helm/helm_package/), optionaly push it to a snapshot repository and uses the following variables:
@@ -288,7 +331,7 @@ If no next version info is generated by `semantic-release`, the package will be
Note: You can disable the `semantic-release` integration described herebefore the `HELM_SEMREL_RELEASE_DISABLED` variable.
-### `helm-publish` job
+#### `helm-publish` job
This job push helm package to a release repository and uses the following variables:
--
GitLab