# GitLab CI template for Helm This project implements a generic GitLab CI template for [Helm](https://helm.sh/). ## Usage In order to include this template in your project, add the following to your `gitlab-ci.yml`: ```yaml include: - project: 'to-be-continuous/helm' ref: '3.3.1' file: '/templates/gitlab-ci-helm.yml' ``` ## Understand This chapter introduces key notions and principle to understand how this template works. ### Managed deployment environments This template implements continuous delivery/continuous deployment based on [Helm](https://helm.sh/) for projects hosted on [Kubernetes](https://kubernetes.io) platforms. It provides several features, usable in different modes (by configuration). #### Review environments The template supports **review** environments: those are dynamic and ephemeral environments to deploy your _ongoing developments_ (a.k.a. _feature_ or _topic_ branches). When enabled, it deploys the result from upstream build stages to a dedicated and temporary environment. It is only active for non-production, non-integration branches. It is a strict equivalent of GitLab's [Review Apps](https://docs.gitlab.com/ee/ci/review_apps/) feature. It also comes with a _cleanup_ job (accessible either from the _environments_ page, or from the pipeline view). #### Integration environment If you're using a Git Workflow with an integration branch (such as [Gitflow](https://www.atlassian.com/git/tutorials/comparing-workflows/gitflow-workflow)), the template supports an **integration** environment. When enabled, it deploys the result from upstream build stages to a dedicated environment. It is only active for your integration branch (`develop` by default). #### Production environments Lastly, the template supports 2 environments associated to your production branch (`master` by default): * a **staging** environment (an iso-prod environment meant for testing and validation purpose), * the **production** environment. You're free to enable whichever or both, and you can also choose your deployment-to-production policy: * **continuous deployment**: automatic deployment to production (when the upstream pipeline is successful), * **continuous delivery**: deployment to production can be triggered manually (when the upstream pipeline is successful). <!-- ### Supported authentication methods --> ### Deployment context variables In order to manage the various deployment environments, this template provides a couple of **dynamic variables** that you might use in your hook scripts and Helm charts (as [values](https://helm.sh/docs/chart_best_practices/values/)): | environment variable | template directive | description | |----------------------|--------------------|-------------| | `$environment_name` | `{{ .Release.Name }}` | a generated application name to use for the current deployment environment (ex: `myproject-review-fix-bug-12` or `myproject-staging`). This is used as the **Helm release name** in deploy & delete jobs - _details below_ | | `$environment_type` | `{{ .Values.environment_type }}` | the current deployment environment type (`review`, `integration`, `staging` or `production`) | | `$hostname` | `{{ .Values.hostname }}` | the environment hostame, if you [specified the environment url statically](#environments-url-management) | #### Generated environment name The `${environment_name}` variable is generated to designate each deployment environment with a unique and meaningful application name. By construction, it is suitable for inclusion in DNS, URLs, Kubernetes labels... It is built from: * the application _base name_ (defaults to `$CI_PROJECT_NAME` but can be overridden globally and/or per deployment environment - _see configuration variables_) * GitLab predefined `$CI_ENVIRONMENT_SLUG` variable ([sluggified](https://en.wikipedia.org/wiki/Clean_URL#Slug) name, truncated to 24 characters) The `${environment_name}` variable is then evaluated as: * `<app base name>` for the production environment * `<app base name>-$CI_ENVIRONMENT_SLUG` for all other deployment environments * :bulb: `${environment_name}` can also be overriden per environment with the appropriate configuration variable Examples (with an application's base name `myapp`): | `$environment_type` | Branch | `$CI_ENVIRONMENT_SLUG` | `$environment_name` | |---------------------|---------------|-------------------------|---------------------| | `review` | `feat/blabla` | `review-feat-bla-xmuzs6`| `myapp-review-feat-bla-xmuzs6` | | `integration` | `develop` | `integration` | `myapp-integration` | | `staging` | `main` | `staging` | `myapp-staging` | | `production` | `main` | `production` | `myapp` | ### Deployment and cleanup scripts The Helm template requires you to provide a Helm chart (either in the project or located in an external repository) to deploy and delete the application. The environment deployment is processed as follows: 1. _optionally_ executes the `helm-pre-deploy.sh` script in your project to perform specific environment pre-initialization (for e.g. create required services), 2. [`helm upgrade`](https://helm.sh/docs/helm/helm_upgrade/) the chart with the configured parameters, using [`$environment_name`](#using-variables) as release name, 3. _optionally_ executes the `helm-post-deploy.sh` script in your project to perform specific environment post-initialization stuff, The environment deletion is processed as follows: 1. _optionally_ executes the `helm-pre-delete.sh` script in your project to perform specific environment pre-cleanup stuff, 2. [`helm uninstall`](https://helm.sh/docs/helm/helm_uninstall/), using [`$environment_name`](#using-variables) as release name, 3. _optionally_ executes the `helm-post-delete.sh` script in your project to perform specific environment post-cleanup (for e.g. delete bound services). :warning: each of the above hook scripts needs to be executable, you can add flag execution with: `git update-index --chmod=+x helm-pre-cleanup.sh` ### Environments URL management The Helm template supports two ways of providing your environments url: * a **static way**: when the environments url can be determined in advance, probably because you're exposing your routes through a DNS you manage, * a [**dynamic way**](https://docs.gitlab.com/ee/ci/environments/#set-dynamic-environment-urls-after-a-job-finishes): when the url cannot be known before the deployment job is executed. The **static way** can be implemented simply by setting the appropriate configuration variable(s) depending on the environment (see environments configuration chapters): * `$HELM_ENVIRONMENT_URL` to define a default url pattern for all your envs, * `$HELM_REVIEW_ENVIRONMENT_URL`, `$HELM_INTEG_ENVIRONMENT_URL`, `$HELM_STAGING_ENVIRONMENT_URL` and `$HELM_PROD_ENVIRONMENT_URL` to override the default. > :information_source: Each of those variables support a **late variable expansion mechanism** with the `%{somevar}` syntax, > allowing you to use any dynamically evaluated variables such as `${environment_name}`. > > Example: > > ```yaml > variables: > HELM_BASE_APP_NAME: "wonderapp" > # global url for all environments > HELM_ENVIRONMENT_URL: "https://%{environment_name}.nonprod.acme.domain" > # override for prod (late expansion of $HELM_BASE_APP_NAME not needed here) > HELM_PROD_ENVIRONMENT_URL: "https://$HELM_BASE_APP_NAME.acme.domain" > # override for review (using separate resource paths) > HELM_REVIEW_ENVIRONMENT_URL: "https://wonderapp-review.nonprod.acme.domain/%{environment_name}" > ``` To implement the **dynamic way**, your deployment script shall simply generate a `environment_url.txt` file in the working directory, containing only the dynamically generated url. When detected by the template, it will use it as the newly deployed environment url. ### Deployment output variables Each deployment job produces _output variables_ that are propagated to downstream jobs (using [dotenv artifacts](https://docs.gitlab.com/ee/ci/pipelines/job_artifacts.html#artifactsreportsdotenv)): * `$environment_type`: set to the type of environment (`review`, `integration`, `staging` or `production`), * `$environment_name`: the application name (see below), * `$environment_url`: set to the environment URL (whether determined statically or dynamically). Those variables may be freely used in downstream jobs (for instance to run acceptance tests against the latest deployed environment). ## Configuration reference ### Secrets management Here are some advices about your **secrets** (variables marked with a :lock:): 1. Manage them as [project or group CI/CD variables](https://docs.gitlab.com/ee/ci/variables/#add-a-cicd-variable-to-a-project): * [**masked**](https://docs.gitlab.com/ee/ci/variables/#mask-a-cicd-variable) to prevent them from being inadvertently displayed in your job logs, * [**protected**](https://docs.gitlab.com/ee/ci/variables/#protected-cicd-variables) if you want to secure some secrets you don't want everyone in the project to have access to (for instance production secrets). 2. In case a secret contains [characters that prevent it from being masked](https://docs.gitlab.com/ee/ci/variables/#mask-a-cicd-variable), simply define its value as the [Base64](https://en.wikipedia.org/wiki/Base64) encoded value prefixed with `@b64@`: it will then be possible to mask it and the template will automatically decode it prior to using it. 3. Don't forget to escape special characters (ex: `$` -> `$$`). ### Global configuration The Helm template uses some global configuration used throughout all jobs. | Name | description | default value | | --------------------- | -------------------------------------- | ----------------- | | `HELM_CLI_IMAGE` | The Docker image used to run Helm <br/>:warning: **set the version required by your Kubernetes server** | `alpine/helm:latest` | | `HELM_CHART_DIR` | The folder where the Helm chart is located | `.` _(root project dir)_ | | `HELM_SCRIPTS_DIR` | The folder where hook scripts are located | `.` _(root project dir)_ | | `HELM_COMMON_VALUES` | Common values file (used for all environments, overridden by specific per-env values files) | undefined (none) | | `HELM_ENV_VALUE_NAME` | The environment type variable set to helm | `environment_type` | | `HELM_HOSTNAME_VALUE_NAME` | The hostname variable set to helm | `hostname` | | `KUBE_NAMESPACE` | The default Kubernetes namespace to use | _none_ but this variable is automatically set by [GitLab Kubernetes integration](https://docs.gitlab.com/ee/user/project/clusters/index.html) when enabled | | :lock: `HELM_DEFAULT_KUBE_CONFIG` | The default kubeconfig content to use | `$KUBECONFIG` (thus supports the [GitLab Kubernetes integration](https://docs.gitlab.com/ee/user/project/clusters/index.html) when enabled) | | `HELM_DEPLOY_ARGS` | The Helm [command with options](https://helm.sh/docs/helm/helm_upgrade/) to deploy the application (_without dynamic arguments such as release name and chart_) | `upgrade --install --atomic --timeout 120s` | | `HELM_DELETE_ARGS` | The Helm [command with options](https://helm.sh/docs/helm/helm_uninstall/) to cleanup the application (_without dynamic arguments such as release name_) | `uninstall` | | `HELM_DEPLOY_CHART` | The Helm [chart](https://helm.sh/docs/topics/charts/) to deploy. _Only required if you want to deploy an **external** chart._ | _none_ | | `HELM_REPOS` | The Helm [chart repositories](https://helm.sh/docs/topics/chart_repository/) to use (formatted as `repo_name_1@:repo_url_1 repo_name_2@:repo_url_2 ...`) | `stable@https://charts.helm.sh/stable bitnami@https://charts.bitnami.com/bitnami` | | `HELM_BASE_APP_NAME` | Base application name | `$CI_PROJECT_NAME` ([see GitLab doc](https://docs.gitlab.com/ee/ci/variables/predefined_variables.html)) | | `HELM_ENVIRONMENT_URL` | Default environments url _(only define for static environment URLs declaration)_<br/>_supports late variable expansion (ex: `https://%{environment_name}.helm.acme.com`)_ | _none_ | ### Review environments configuration Review environments are dynamic and ephemeral environments to deploy your _ongoing developments_ (a.k.a. _feature_ or _topic_ branches). They are **enabled by default** and can be disabled by setting the `HELM_REVIEW_DISABLED` variable (see below). Here are variables supported to configure review environments: | Name | description | default value | | ------------------------ | -------------------------------------- | ----------------- | | `HELM_REVIEW_DISABLED` | Set to `true` to disable `review` env | _none_ (enabled) | | `HELM_REVIEW_APP_NAME` | Application name for `review` env | `"${HELM_BASE_APP_NAME}-${CI_ENVIRONMENT_SLUG}"` (ex: `myproject-review-fix-bug-12`) | | `HELM_REVIEW_ENVIRONMENT_URL`| The review environments url _(only define for static environment URLs declaration and if different from default)_ | `$HELM_ENVIRONMENT_URL` | | `HELM_REVIEW_NAMESPACE` | The Kubernetes namespace to use for `review` env _(only define to override default)_ | `$KUBE_NAMESPACE` | | :lock: `HELM_REVIEW_KUBE_CONFIG` | kubeconfig content used for `review` env _(only define to override default)_ | `$HELM_DEFAULT_KUBE_CONFIG` | | `HELM_REVIEW_VALUES` | The [Values file](https://helm.sh/docs/chart_template_guide/values_files/) to use with `review` environments | _none_ | ### Integration environment configuration The integration environment is the environment associated to your integration branch (`develop` by default). It is **enabled by default** and can be disabled by setting the `HELM_INTEG_DISABLED` variable (see below). Here are variables supported to configure the integration environment: | Name | description | default value | | ------------------------ | -------------------------------------- | ----------------- | | `HELM_INTEG_DISABLED` | Set to `true` to disable `integration` env | _none_ (enabled) | | `HELM_INTEG_APP_NAME` | Application name for `integration` env | `$HELM_BASE_APP_NAME-integration` | | `HELM_INTEG_ENVIRONMENT_URL`| The integration environment url _(only define for static environment URLs declaration and if different from default)_ | `$HELM_ENVIRONMENT_URL` | | `HELM_INTEG_NAMESPACE` | The Kubernetes namespace to use for `integration` env _(only define to override default)_ | `$KUBE_NAMESPACE` | | :lock: `HELM_INTEG_KUBE_CONFIG` | kubeconfig content used for `integration` env _(only define to override default)_ | `$HELM_DEFAULT_KUBE_CONFIG` | | `HELM_INTEG_VALUES` | The [Values file](https://helm.sh/docs/chart_template_guide/values_files/) to use with the `integration` environment | _none_ | ### Staging environment configuration The staging environment is an iso-prod environment meant for testing and validation purpose associated to your production branch (`master` by default). It is **enabled by default** and can be disabled by setting the `HELM_STAGING_DISABLED` variable (see below). Here are variables supported to configure the staging environment: | Name | description | default value | | ------------------------ | -------------------------------------- | ----------------- | | `HELM_STAGING_DISABLED` | Set to `true` to disable `staging` env | _none_ (enabled) | | `HELM_STAGING_APP_NAME` | Application name for `staging` env | `$HELM_BASE_APP_NAME-staging` | | `HELM_STAGING_ENVIRONMENT_URL`| The staging environment url _(only define for static environment URLs declaration and if different from default)_ | `$HELM_ENVIRONMENT_URL` | | `HELM_STAGING_NAMESPACE` | The Kubernetes namespace to use for `staging` env _(only define to override default)_ | `$KUBE_NAMESPACE` | | :lock: `HELM_STAGING_KUBE_CONFIG` | kubeconfig content used for `staging` env _(only define to override default)_ | `$HELM_DEFAULT_KUBE_CONFIG` | | `HELM_STAGING_VALUES` | The [Values file](https://helm.sh/docs/chart_template_guide/values_files/) to use with the staging environment | _none_ | ### Production environment configuration The production environment is the final deployment environment associated with your production branch (`master` by default). It is **enabled by default** and can be disabled by setting the `HELM_PROD_DISABLED` variable (see below). Here are variables supported to configure the production environment: | Name | description | default value | | ------------------------ | -------------------------------------- | ----------------- | | `HELM_PROD_DISABLED` | Set to `true` to disable `production` env | _none_ (enabled) | | `HELM_PROD_APP_NAME` | Application name for `production` env | `$HELM_BASE_APP_NAME` | | `HELM_PROD_ENVIRONMENT_URL`| The production environment url _(only define for static environment URLs declaration and if different from default)_ | `$HELM_ENVIRONMENT_URL` | | `HELM_PROD_NAMESPACE` | The Kubernetes namespace to use for `production` env _(only define to override default)_ | `$KUBE_NAMESPACE` | | :lock: `HELM_PROD_KUBE_CONFIG` | kubeconfig content used for `production` env _(only define to override default)_ | `$HELM_DEFAULT_KUBE_CONFIG` | | `AUTODEPLOY_TO_PROD` | Set this variable to auto-deploy to production. If not set deployment to production will be `manual` (default behaviour). | _none_ (disabled) | | `HELM_PROD_VALUES` | The [Values file](https://helm.sh/docs/chart_template_guide/values_files/) to use with the production environment | _none_ | ### `helm-lint` job This job [examines your chart for possible issues](https://helm.sh/docs/helm/helm_lint/) and uses the following variables: | Name | description | default value | | --------------------- | ---------------------------------------- | ----------------- | | `HELM_LINT_DISABLED` | Set to `true` to disable Helm lint | _none_ (enabled) | | `HELM_LINT_ARGS` | The Helm [command with options](https://helm.sh/docs/helm/helm_lint/) to trigger the analysis (_without dynamic arguments such as the chart path_) | `lint --strict` | | `HELM_DEPENDENCY_ARGS` | The Helm [command with options](https://helm.sh/docs/helm/helm_dependency_update/) to update on-disk the chart dependencies (_without dynamic arguments such as the chart path_) | `dependency update` | ### `helm-values-*-lint` job These jobs perform a [Yaml Lint](https://github.com/adrienverge/yamllint) of your Helm [values file](https://helm.sh/docs/chart_template_guide/values_files/) and uses the following variables: | Name | description | default value | | ------------------------ | ------------------------------------- | ----------------- | | `HELM_YAMLLINT_IMAGE` | The Docker image used to run YamlLint test | `cytopia/yamllint` | | `HELM_YAMLLINT_DISABLED` | Set to `true` to disable Yaml lint | _none_ (enabled) | | `HELM_YAMLLINT_CONFIG` | Config used with the yamllint tool | `{extends: relaxed, rules: {line-length: {max: 160}}}` | | `HELM_YAMLLINT_ARGS` | Arguments used by the lint job | `-f colored --strict` | ### `helm-*-score` job This job runs [Kube-Score](https://kube-score.com/) on the resources to be created by Helm and uses the following variables: | Name | description | default value | | --------------------- | ---------------------------------------- | ----------------- | | `HELM_KUBE_SCORE_DISABLED` | Set to `true` to disable [Kube-Score](https://kube-score.com/) | _none_ (enabled) | | `HELM_KUBE_SCORE_IMAGE` | The Docker image used to run [Kube-Score](https://kube-score.com/) | `zegl/kube-score:latest-helm3` | | `HELM_KUBE_SCORE_ARGS` | Arguments used by the helm-score job | _none_ | ### Charts publishing The template builds a chart package that may be pushed as two distinct packages, depending on a certain _workflow_: 1. **snapshot**: the chart is first packaged and then pushed to some registry as the **snapshot** image. It can be seen as the raw result of the build, but still **untested and unreliable**. 2. **release**: once the snapshot chart has been thoroughly tested (both by `package-test` stage jobs and/or `acceptance` stage jobs after being deployed to some server), then the chart is pushed one more time as the **release** chart. This second push can be seen as the **promotion** of the snapshot chart being now **tested and reliable**. Common variables for `helm-package` and `helm-pusblish`: | Name | description | default value | | --------------------------------- | -------------------------------------------- | ----------------------- | | `HELM_REPO_PUBLISH_METHOD` | HTTP method to use to push the package | `POST` | | :lock: `HELM_REPO_USER` | Helm registry username | `$CI_REGISTRY_USER` | | :lock: `HELM_REPO_PASSWORD` | Helm registry password | `$CI_REGISTRY_PASSWORD` | #### `helm-package` job This job [packages your chart into an archive](https://helm.sh/docs/helm/helm_package/), optionaly push it to a snapshot repository and uses the following variables: | Name | description | default value | | ------------------------------------- | ---------------------------------------- | ----------------- | | `HELM_PACKAGE_ARGS` | The Helm [command with options](https://helm.sh/docs/helm/helm_package/) to perform the packaging (_without dynamic arguments such as the chart path_) | `package --dependency-update` | | `HELM_SEMREL_RELEASE_DISABLED` | Set to `true` to disable usage of `semantic-release` release info for helm package (see next chapter) | _none_ (enabled) | | `HELM_PUBLISH_SNAPSHOT_URL` | The URL of the Helm repository to publish your Helm package as a snapshot | _gitlab repository on snapshot channel_ `${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/packages/helm/api/snapshot/charts` | | `HELM_REPO_SNAPSHOT_PUBLISH_METHOD` | HTTP method to use to push the package | `$HELM_REPO_PUBLISH_METHOD` | | `HELM_REPO_SNAPSHOT_USER` | Snapshot repository username | `$HELM_REPO_USER` | | :lock: `HELM_REPO_SNAPSHOT_PASSWORD` | Snapshot repository password | `$HELM_REPO_PASSWORD` | #### `semantic-release` integration If you activate the [`semantic-release-info` job from the `semantic-release` template](https://gitlab.com/to-be-continuous/semantic-release/#semantic-release-info-job), the `helm-publish` job will automatically use the generated next version info for both application version (`--app-version`) and chart version (`--version`). If no next version info is generated by `semantic-release`, the package will be created, but without versioning info. Note: You can disable the `semantic-release` integration described herebefore the `HELM_SEMREL_RELEASE_DISABLED` variable. #### `helm-publish` job This job push helm package to a release repository and uses the following variables: | Name | description | default value | | ----------------------------------- | --------------------------------------------- | --------------------------------- | | `HELM_PUBLISH_URL` | The URL of the Helm repository to publish your Helm package | _gitlab repository on release channel_ `${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/packages/helm/api/release/charts` | | `HELM_REPO_RELEASE_PUBLISH_METHOD` | HTTP method to use to push the package | `$HELM_REPO_PUBLISH_METHOD` | | `HELM_REPO_RELEASE_USER` | Release repository username (for PUT request auth) | `$HELM_REPO_USER` | | :lock: `HELM_REPO_RELEASE_PASSWORD` | Release repository password (for PUT request auth) | `$HELM_REPO_PASSWORD` | ### `helm-test` job This job runs [Helm tests](https://helm.sh/docs/topics/chart_tests/). The job definition must contain the helm test hook annotation: `helm.sh/hook: test` You are welcome to nest your test suite under a `tests/` directory like `$HELM_CHART_DIR/templates/tests/` for more isolation. It is **disabled by default** and can be enabled by setting the ``HELM_TEST_ENABLED`` variable (see below). It uses the following variables: | Name | description | default value | | --------------------- | ---------------------------------------- | ----------------- | | `HELM_TEST_ENABLED` | Set to `true` to enable Helm test | _none_ (disabled) | | `HELM_TEST_ARGS` | The Helm [command with options](https://helm.sh/docs/helm/helm_test/) to perform acceptance test (_without dynamic arguments such as the chart path_) | `test` | ## Variants ### Vault variant This variant allows delegating your secrets management to a [Vault](https://www.vaultproject.io/) server. #### Configuration In order to be able to communicate with the Vault server, the variant requires the additional configuration parameters: | Name | description | default value | | ----------------- | -------------------------------------- | ----------------- | | `TBC_VAULT_IMAGE` | The [Vault Secrets Provider](https://gitlab.com/to-be-continuous/tools/vault-secrets-provider) image to use (can be overridden) | `$CI_REGISTRY/to-be-continuous/tools/vault-secrets-provider:master` | | `VAULT_BASE_URL` | The Vault server base API url | _none_ | | :lock: `VAULT_ROLE_ID` | The [AppRole](https://www.vaultproject.io/docs/auth/approle) RoleID | **must be defined** | | :lock: `VAULT_SECRET_ID` | The [AppRole](https://www.vaultproject.io/docs/auth/approle) SecretID | **must be defined** | #### Usage Then you may retrieve any of your secret(s) from Vault using the following syntax: ```text @url@http://vault-secrets-provider/api/secrets/{secret_path}?field={field} ``` With: | Name | description | | -------------------------------- | -------------------------------------- | | `secret_path` (_path parameter_) | this is your secret location in the Vault server | | `field` (_query parameter_) | parameter to access a single basic field from the secret JSON payload | #### Example ```yaml include: # main template - project: 'to-be-continuous/helm' ref: '3.3.1' file: '/templates/gitlab-ci-helm.yml' # Vault variant - project: 'to-be-continuous/helm' ref: '3.3.1' file: '/templates/gitlab-ci-helm-vault.yml' variables: # Secrets managed by Vault HELM_DEFAULT_KUBE_CONFIG: "@url@http://vault-secrets-provider/api/secrets/b7ecb6ebabc231/my-app/helm/noprod?field=kube_config" HELM_PROD_KUBE_CONFIG: "@url@http://vault-secrets-provider/api/secrets/b7ecb6ebabc231/my-app/helm/prod?field=kube_config" VAULT_BASE_URL: "https://vault.acme.host/v1" # $VAULT_ROLE_ID and $VAULT_SECRET_ID defined as a secret CI/CD variable ```